Data Aggregation in Large Scale Distributed Systems

Data aggregation, in its most basic definition, is the ability to summarize information. Data aggregation is highly relevant to distributed systems, which collect and process information from many sources, like Internet-scale information systems, and peer-to-peer data management systems. Several architectures and techniques have been recently explored (either in the context of Internet services, peer-to-peer, or wireless sensor networks research), aiming at the design of general purpose, highly scalable data aggregation services. In this chapter, we describe application scenarios, and review and compare several proposed designs for aggregation services. In the second part of the chapter, we present a complete case study for aggregation services, in the context of a DHT-based peer data management architecture. We argue that aggregation services would benefit for a design, which is consistent with and integrated into a general purpose data indexing functionality, and present GREG, an architecture which is based on this assumption.