The development of multilingual and multimedia information access systems calls for proper evaluation methodologies to ensure that they meet the expected user requirements and provide the desired effectiveness. IR research offers a strong evaluation methodology and a range of evaluation metrics, such as MAP and (n)DCG. In this paper, we propose a new metric for ranking evaluation, the CRP. We start with the observation that a document of a given degree of relevance may be ranked too early or too late regarding the ideal ranking of documents for a query. Its relative position may be negative, indicating too early ranking, zero indicating correct ranking, or positive, indicating too late ranking. By cumulating these relative rankings we indicate, at each ranked position, the net effect of document displacements, the CRP. We first define the metric formally and then discuss its properties, its relationship to prior metrics, and its visualization. Finally we propose different visualizations of CRP by exploiting a test collection to demonstrate its behavior. © 2012 Springer-Verlag.

Cumulated relative position: A metric for ranking evaluation / Angelini, Marco; Nicola, Ferro; Kalervo, Jarvelin; Heikki, Keskustalo; Ari, Pirkola; Santucci, Giuseppe; Gianmaria, Silvello. - STAMPA. - 7488 LNCS:(2012), pp. 112-123. (Intervento presentato al convegno 3rd International Conference of the CLEF Initiative, CLEF 2012 tenutosi a Rome nel 17 September 2012 through 20 September 2012) [10.1007/978-3-642-33247-0_13].

Cumulated relative position: A metric for ranking evaluation

ANGELINI, MARCO;SANTUCCI, Giuseppe;
2012

Abstract

The development of multilingual and multimedia information access systems calls for proper evaluation methodologies to ensure that they meet the expected user requirements and provide the desired effectiveness. IR research offers a strong evaluation methodology and a range of evaluation metrics, such as MAP and (n)DCG. In this paper, we propose a new metric for ranking evaluation, the CRP. We start with the observation that a document of a given degree of relevance may be ranked too early or too late regarding the ideal ranking of documents for a query. Its relative position may be negative, indicating too early ranking, zero indicating correct ranking, or positive, indicating too late ranking. By cumulating these relative rankings we indicate, at each ranked position, the net effect of document displacements, the CRP. We first define the metric formally and then discuss its properties, its relationship to prior metrics, and its visualization. Finally we propose different visualizations of CRP by exploiting a test collection to demonstrate its behavior. © 2012 Springer-Verlag.
2012
3rd International Conference of the CLEF Initiative, CLEF 2012
04 Pubblicazione in atti di convegno::04b Atto di convegno in volume
Cumulated relative position: A metric for ranking evaluation / Angelini, Marco; Nicola, Ferro; Kalervo, Jarvelin; Heikki, Keskustalo; Ari, Pirkola; Santucci, Giuseppe; Gianmaria, Silvello. - STAMPA. - 7488 LNCS:(2012), pp. 112-123. (Intervento presentato al convegno 3rd International Conference of the CLEF Initiative, CLEF 2012 tenutosi a Rome nel 17 September 2012 through 20 September 2012) [10.1007/978-3-642-33247-0_13].
File allegati a questo prodotto
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11573/463339
 Attenzione

Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo

Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 8
  • ???jsp.display-item.citation.isi??? ND
social impact