In this paper we present a method to evaluate the quality of a rater’s judgement, which can integrate and enrich the use of inter-rater agreement as a reliability measure. Our proposal is an integrative one and evaluates the quality of a rater’s performance through an analysis of the profile of that individual rater’s performance. We discuss its rationale on the basis of the interpretation of inter-rater agreement, highlighting some critical issues. For this purpose, we adopt a computational model based on fuzzy set theory, demonstrating its main characteristics with an exemplary case study.
A fuzzy set theory based computational model to represent the quality of inter-rater agreement / Ciavolino, E.; Salvatore, S.; Calcagnì, A.. - In: QUALITY & QUANTITY. - ISSN 0033-5177. - 48(4):(2014), pp. 2225-2240. [10.1007/s11135-013-9888-3]
A fuzzy set theory based computational model to represent the quality of inter-rater agreement
S. Salvatore;
2014
Abstract
In this paper we present a method to evaluate the quality of a rater’s judgement, which can integrate and enrich the use of inter-rater agreement as a reliability measure. Our proposal is an integrative one and evaluates the quality of a rater’s performance through an analysis of the profile of that individual rater’s performance. We discuss its rationale on the basis of the interpretation of inter-rater agreement, highlighting some critical issues. For this purpose, we adopt a computational model based on fuzzy set theory, demonstrating its main characteristics with an exemplary case study.I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
