RoboCup@Home: Adaptive benchmarking of robot bodies and minds

RoboCup@Home is the largest benchmarking effort for domestic service robots. The benchmarking is in the form of a competition, with several yearly local competitions and an international one. Every year the tests become more complex, depending on the results of the previous years. In the past four years the focus has been on benchmarking physical aspects of the robots, such as manipulation, recognizing people and human-robot interaction. In 2010, for the first time, there is a test which is targeted at the mental cognitive capabilities of the robot. In order to guarantee scientific quality of the proposed solutions and effective integration in a fully working system, all the tests include different capabilities and change every year. This novel feature of RoboCup@Home benchmarking raises the question of: How can effective benchmark tests be defined and at the same time measure the progress over many years? In this paper we present the methodology applied in and results of RoboCup@Home for measuring the effectiveness of benchmarking service robots through competitions and present a new integrated test for benchmarking the cognitive abilities of a robot. © 2011 Springer-Verlag.