Latent Block Models are used for probabilistic biclustering, which is shown to be an effective method for analyzing various relational data sets. However, there has been no statistical test method for determining the row and column cluster numbers of Latent Block Models. Recent studies have constructed statistical-test-based methods for Stochastic Block Models, in which we assume that the observed matrix is a square symmetric matrix and that the cluster assignments are the same for rows and columns. In this paper, we develop a goodness-of-fit test for Latent Block Models, which tests whether an observed data matrix fits a given set of row and column cluster numbers, or it consists of more clusters in at least one direction of row and column. To construct the test method, we use a result from random matrix theory for a sample covariance matrix. We show experimentally the effectiveness of our proposed method, by showing the asymptotic behavior of the test statistic and the test accuracy.