Human labeling errors and their impact on ConvNets for satellite image scene classification

Convolutional neural networks (ConvNets) have been successfully applied to satellite image scene classification. Human-labeled training datasets are essential for ConvNets to perform accurate classification. Errors in human-labeled training datasets are unavoidable due to the complexity of satellite images. However, the distribution of human labeling errors on satellite images and their impact on ConvNets have not been investigated. To fill this research gap, this study, for the first time, collected real-world labels from 32 participants and explored how their errors affect three ConvNets (VGG16, GoogleNet and ResNet-50) for high-resolution satellite image scene classification. We found that: (1) human labeling errors have significant class and instance dependence, which is fundamentally different from the simulation noise in previous studies; (2) regarding the overall accuracy of all classes, when human labeling errors in training data increase by one unit, the overall accuracy of ConvNets classification decreases by approximately half a unit; (3) regarding the accuracy of each class, the impact of human labeling errors on ConvNets shows large heterogeneity across classes. To uncover the mechanism underlying the impact of human labeling errors on ConvNets, we further compared it with two types of simulated labeling noise: uniform noise (errors independent of both classes and instances) and class-dependent noise (errors independent of instances but not classes). Our results show that the impact of human labeling errors on ConvNets is similar to that of the simulated class-dependent noise but not to that of the simulated uniform noise, suggesting that the impact of human labeling errors on ConvNets is mainly due to class-dependent errors rather than instance-dependent errors.