Harnessing the scientific data produced by the experimental evaluation of search engines and information access systems

Measuring is a key to scientific progress. This is particularly true for research concerning complex systems, whether natural or human-built. Multilingual and multimedia information systems are increasingly complex: they need to satisfy diverse user needs and support challenging tasks. Their development calls for proper evaluation methodologies to ensure that they meet the expected user requirements and provide the desired effectiveness. In the process, vast amounts of experimental data are generated that beg for analysis tools to enable interpretation and thereby facilitate scientific and technological progress. These scientific data are at the basis of the research and publications in the field and can be better exploited and linked to the scientific literature and production. We are building a software infrastructure, called Distributed Information Retrieval Evaluation Campaign Tool (DIRECT), to manage, curate, enrich, and make these scientific data online accessible and we discuss how this infrastructure can be exploited to directly link the experimental data into papers and publications describing them. © 2011 Published by Elsevier Ltd.