Assessing Inter- and Intra-Rater Reliability in Multi-Layer Annotations: A study on Tense and Aspect in learners of English as L3

This paper reports on the methodological procedures implemented to address issues of reliability in manual annotations as part of an ongoing PhD project on the Acquisition of Tense and Aspect in learners of English as L3. A combination of multiple procedures to assess inter- and intra-rater reliability coefficients among several coders proved to be a useful method to reflect on possible cases of low agreement in problematic layers of annotations and to highlight emerging annotation patterns among the coders. Moreover, the tests designed in our study aided in identifying potential problems associated with developing instruments and guidelines that explained the coding task, pinpointing particularly challenging categories, and contributing to improving overall accuracy and transparency in the annotation process.