When Annotators Disagree: A Principled Approach to Learning with Noisy Labels

In practical settings, classification datasets are often labeled by humans, leading to potential noise due to varying annotations from different individuals. The exact noise distribution impacting these labels is typically unknown; however, one quantity we can measure and attempt to exploit is inter-rater agreement. Building on this, our work makes key contributions: we (i) demonstrate how inter-annotator statistics can be used to estimate the label noise distribution; (ii) propose methods that leverage these estimates to train models on noisy data; and (iii) derive generalization bounds within the empirical risk minimization framework that depend on the estimated noise characteristics. Finally, we present experiments that support our findings.