Research assessment exercises have now become common evaluation tools in a number of countries. These exercises have the goal of guiding merit-based public funds allocation, stimulating improvement of research productivity through competition and assessing the impact of adopted research support policies. One case in point is Italy's most recent research assessment effort, VQR 2011–2014 (Research Quality Evaluation), which, in addition to research institutions, also evaluated university departments, and individuals in some cases (i.e., recently hired research staff and members of PhD committees). However, the way an institution's score was divided, according to VQR rules, between its constituent departments or its staff members does not enjoy many desirable properties well known from coalitional game theory (e.g., budget balance, fairness, marginality). We propose, instead, an alternative score division rule that is based on the notion of Shapley value, a well known solution concept in coalitional game theory, which enjoys the desirable properties mentioned above. For a significant test case (namely, Sapienza University of Rome, the largest university in Italy), we present a detailed comparison of the scores obtained, for substructures and individuals, by applying the official VQR rules, with those resulting from Shapley value computations. We show that there are significant differences in the resulting scores, making room for improvements in the allocation rules used in research assessment exercises.

On the Shapley value and its application to the Italian VQR research assessment exercise / Demetrescu, Camil; Lupia, Francesco; Mendicelli, Angelo; Ribichini, Andrea; Scarcello, Francesco; Schaerf, Marco. - In: JOURNAL OF INFORMETRICS. - ISSN 1751-1577. - 13:1(2019), pp. 87-104. [10.1016/j.joi.2018.11.008]

On the Shapley value and its application to the Italian VQR research assessment exercise

Demetrescu, Camil;Ribichini, Andrea;Schaerf, Marco
2019

Abstract

Research assessment exercises have now become common evaluation tools in a number of countries. These exercises have the goal of guiding merit-based public funds allocation, stimulating improvement of research productivity through competition and assessing the impact of adopted research support policies. One case in point is Italy's most recent research assessment effort, VQR 2011–2014 (Research Quality Evaluation), which, in addition to research institutions, also evaluated university departments, and individuals in some cases (i.e., recently hired research staff and members of PhD committees). However, the way an institution's score was divided, according to VQR rules, between its constituent departments or its staff members does not enjoy many desirable properties well known from coalitional game theory (e.g., budget balance, fairness, marginality). We propose, instead, an alternative score division rule that is based on the notion of Shapley value, a well known solution concept in coalitional game theory, which enjoys the desirable properties mentioned above. For a significant test case (namely, Sapienza University of Rome, the largest university in Italy), we present a detailed comparison of the scores obtained, for substructures and individuals, by applying the official VQR rules, with those resulting from Shapley value computations. We show that there are significant differences in the resulting scores, making room for improvements in the allocation rules used in research assessment exercises.
2019
Research assessment; Bibliometrics;; Shapley value; Research productivity
01 Pubblicazione su rivista::01a Articolo in rivista
On the Shapley value and its application to the Italian VQR research assessment exercise / Demetrescu, Camil; Lupia, Francesco; Mendicelli, Angelo; Ribichini, Andrea; Scarcello, Francesco; Schaerf, Marco. - In: JOURNAL OF INFORMETRICS. - ISSN 1751-1577. - 13:1(2019), pp. 87-104. [10.1016/j.joi.2018.11.008]
File allegati a questo prodotto
File Dimensione Formato  
Demetrescu_On-the-Shapley_2019.pdf

solo gestori archivio

Tipologia: Versione editoriale (versione pubblicata con il layout dell'editore)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 1.36 MB
Formato Adobe PDF
1.36 MB Adobe PDF   Contatta l'autore
Demetrescu_Preprint_On-the-Shapley_2019.pdf

accesso aperto

Note: https://www.sciencedirect.com/science/article/pii/S1751157718302141
Tipologia: Documento in Pre-print (manoscritto inviato all'editore, precedente alla peer review)
Licenza: Creative commons
Dimensione 906.53 kB
Formato Adobe PDF
906.53 kB Adobe PDF

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11573/1204366
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 11
  • ???jsp.display-item.citation.isi??? 9
social impact