Subjective tests for the assessment of the quality of experience (QoE) are typically run with a pool of subjects providing their opinion score using a 5-level scale. The subjects? Mean Opinion Score (MOS) is generally assumed as the best estimation of the average score in the target population. Indeed, for a large enough sample we can assume that the mean of the variations across the subjects approaches zero, but this is not the case for the limited number of subjects typically considered in subjective tests. In this paper we propose a model for the estimation of the population average QoE. We apply such model to a dataset composed of the individual scores assigned by 25 subjects to a set of gaming videos evaluated under different resolutions and compression rates. The model recognizes the ordinal multinomial nature of the data and allows for correlation between scores of the same subject on different data. The resulting estimated average QoE is shown to follow more credible patterns than the MOS, in particular with respect to improved compression rates, for which model estimates present a more coherent behaviour. In order to favour reproducibility and application for different datasets, the software that implements the model is also made publicly available.
Estimation of Quality Scores from Subjective Tests - beyond Subjects' MOS / Pezzulli, Sergio; Martini, Maria G.; Barman, Nabajeet. - In: IEEE TRANSACTIONS ON MULTIMEDIA. - ISSN 1520-9210. - (2020), pp. 1-16. [10.1109/tmm.2020.3013349]
Estimation of Quality Scores from Subjective Tests - beyond Subjects' MOS
Sergio Pezzulli
Primo
Membro del Collaboration Group
;
2020
Abstract
Subjective tests for the assessment of the quality of experience (QoE) are typically run with a pool of subjects providing their opinion score using a 5-level scale. The subjects? Mean Opinion Score (MOS) is generally assumed as the best estimation of the average score in the target population. Indeed, for a large enough sample we can assume that the mean of the variations across the subjects approaches zero, but this is not the case for the limited number of subjects typically considered in subjective tests. In this paper we propose a model for the estimation of the population average QoE. We apply such model to a dataset composed of the individual scores assigned by 25 subjects to a set of gaming videos evaluated under different resolutions and compression rates. The model recognizes the ordinal multinomial nature of the data and allows for correlation between scores of the same subject on different data. The resulting estimated average QoE is shown to follow more credible patterns than the MOS, in particular with respect to improved compression rates, for which model estimates present a more coherent behaviour. In order to favour reproducibility and application for different datasets, the software that implements the model is also made publicly available.File | Dimensione | Formato | |
---|---|---|---|
Pezzulli_Estimation-Quality-Scores_2020.pdf
solo gestori archivio
Tipologia:
Versione editoriale (versione pubblicata con il layout dell'editore)
Licenza:
Tutti i diritti riservati (All rights reserved)
Dimensione
1.8 MB
Formato
Adobe PDF
|
1.8 MB | Adobe PDF | Contatta l'autore |
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.