Label distribution learning (LDL) trains a model to predict the relevance of a set of labels (called label distribution (LD)) to an instance. The previous LDL methods all assumed the LDs of the training instances are accurate. However, annotating highly accurate LDs for training instances is time-consuming and very expensive, and in reality the collected LD is usually inaccurate and disturbed by annotating errors. For the first time, this paper investigates the problem of inaccurate LDL, i.e., developing an LDL model with noisy LDs. Specifically, we assume the noisy LD matrix is the linear combination of an ideal LD matrix and a sparse noisy matrix. Accordingly, inaccurate LDL becomes an inverse problem, i.e., recovering the ideal LD and noise matrix from the inaccurate LDs. To this end, we assume the ideal LD matrix is low-rank due to the correlation of labels. Besides, we use the local geometric structure of instances captured by a graph to assist the ideal LD recovery as if two instances are similar to each other, they are likely to share the same LD. The proposed model is finally formulated as a graph-regularized low-rank and sparse decomposition problem and numerically solved by the alternating direction method of multipliers. Extensive experiments demonstrate that our method can recover a relatively accurate LD from the inaccurate LD and promote the performance of different LDL methods with inaccurate LD.