Measuring is a key to scientific progress. This is particularly true for research concerning complex systems, whether natural or humanbuilt. Multilingual and multimedia information access systems, such as search engines, are increasingly complex: they need to satisfy diverse user needs and support challenging tasks. Their development calls for proper evaluation methodologies to ensure that they meet the expected user requirements and provide the desired effectiveness. In this context, failure analysis is crucial to understand the behaviour of complex systems. Unfortunately, this is an especially challenging activity, requiring vast amounts of human effort to inspect query-by-query the output of a system in order to understand what went well or bad. It is therefore fundamental to provide automated tools to examine system behaviour, both visually and analytically. Moreover, once you understand the reason behind a failure, you still need to conduct a "what-if" analysis to understand what among the different possible solutions is most promising and effective before actually starting to modify your system. This paper provides an analytical model for examining performances of IR systems, based on the discounted cumulative gain family of metrics, and visualization for interacting and exploring the performances of the system under examination. Moreover, we propose machine learning approach to learn the ranking model of the examined system in order to be able to conduct a "what-if" analysis and visually explore what can happen if you adopt a given solution before having to actually implement it. Copyright © 2012 ACM.

Visual interactive failure analysis: Supporting users in information retrieval evaluation / Angelini, Marco; Nicola, Ferro; Santucci, Giuseppe; Gianmaria, Silvello. - STAMPA. - (2012), pp. 194-203. (Intervento presentato al convegno 4th Information Interaction in Context Symposium, IIiX 2012 tenutosi a Nijmegen nel 21 August 2012 through 24 August 2012) [10.1145/2362724.2362757].

Visual interactive failure analysis: Supporting users in information retrieval evaluation

ANGELINI, MARCO;SANTUCCI, Giuseppe;
2012

Abstract

Measuring is a key to scientific progress. This is particularly true for research concerning complex systems, whether natural or humanbuilt. Multilingual and multimedia information access systems, such as search engines, are increasingly complex: they need to satisfy diverse user needs and support challenging tasks. Their development calls for proper evaluation methodologies to ensure that they meet the expected user requirements and provide the desired effectiveness. In this context, failure analysis is crucial to understand the behaviour of complex systems. Unfortunately, this is an especially challenging activity, requiring vast amounts of human effort to inspect query-by-query the output of a system in order to understand what went well or bad. It is therefore fundamental to provide automated tools to examine system behaviour, both visually and analytically. Moreover, once you understand the reason behind a failure, you still need to conduct a "what-if" analysis to understand what among the different possible solutions is most promising and effective before actually starting to modify your system. This paper provides an analytical model for examining performances of IR systems, based on the discounted cumulative gain family of metrics, and visualization for interacting and exploring the performances of the system under examination. Moreover, we propose machine learning approach to learn the ranking model of the examined system in order to be able to conduct a "what-if" analysis and visually explore what can happen if you adopt a given solution before having to actually implement it. Copyright © 2012 ACM.
2012
4th Information Interaction in Context Symposium, IIiX 2012
best practices; data test collection; evaluation infrastructure; experimental evaluation; scientific data
04 Pubblicazione in atti di convegno::04b Atto di convegno in volume
Visual interactive failure analysis: Supporting users in information retrieval evaluation / Angelini, Marco; Nicola, Ferro; Santucci, Giuseppe; Gianmaria, Silvello. - STAMPA. - (2012), pp. 194-203. (Intervento presentato al convegno 4th Information Interaction in Context Symposium, IIiX 2012 tenutosi a Nijmegen nel 21 August 2012 through 24 August 2012) [10.1145/2362724.2362757].
File allegati a questo prodotto
File Dimensione Formato  
VE_2012_11573-503181.pdf

solo gestori archivio

Tipologia: Versione editoriale (versione pubblicata con il layout dell'editore)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 1.1 MB
Formato Adobe PDF
1.1 MB Adobe PDF   Contatta l'autore

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11573/503181
 Attenzione

Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo

Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 9
  • ???jsp.display-item.citation.isi??? ND
social impact