Clearing noisy annotations for computed tomography imaging

One of the problems on the way to successful implementation of neural networks is the quality of annotation. For instance, different annotators can annotate images in a different way and very often their decisions do not match exactly and in extreme cases are even mutually exclusive which results in noisy annotations and, consequently, inaccurate predictions. To avoid that problem in the task of computed tomography (CT) imaging segmentation we propose a clearing algorithm for annotations. It consists of 3 stages: - annotators scoring, which assigns a higher confidence level to better annotators; - nodules scoring, which assigns a higher confidence level to nodules confirmed by good annotators; - nodules merging, which aggregates annotations according to nodules confidence. In general, the algorithm can be applied to many different tasks (namely, binary and multi-class semantic segmentation, and also with trivial adjustments to classification and regression) where there are several annotators labeling each image.