Distributed stream processing frameworks are designed to perform continuous computation on possibly unbounded data streams whose rates can change over time. Devising solutions to make such systems elastically scale is a fundamental goal to achieve desired performance and cut costs caused by resource over-provisioning. These systems can be scaled along two dimensions: the operator parallelism and the number of resources. In this paper, we show how these two dimensions, as two symbiotic entities, are independent but must mutually interact for the global benefit of the system. On the basis of this observation, we propose a fine-grained model for estimating the resource utilization of a stream processing application that enables the independent scaling of operators and resources. A simple, yet effective, combined management of the two dimensions allows us to propose ELYSIUM, a novel elastic scaling approach that provides efficient resource utilization. We implemented the proposed approach within Apache Storm and tested it by running two real-world applications with different input load curves. The outcomes backup our claims showing that the proposed symbiotic management outperforms elastic scaling strategies where operators and resources are jointly scaled.

Elastic Symbiotic Scaling of Operators and Resources in Stream Processing Systems / Lombardi, Federico; Aniello, Leonardo; Bonomi, Silvia; Querzoni, Leonardo. - In: IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS. - ISSN 1045-9219. - STAMPA. - 29:3(2018), pp. 572-585. [10.1109/TPDS.2017.2762683]

Elastic Symbiotic Scaling of Operators and Resources in Stream Processing Systems

Federico Lombardi;Leonardo Aniello;Silvia Bonomi;Leonardo Querzoni
2018

Abstract

Distributed stream processing frameworks are designed to perform continuous computation on possibly unbounded data streams whose rates can change over time. Devising solutions to make such systems elastically scale is a fundamental goal to achieve desired performance and cut costs caused by resource over-provisioning. These systems can be scaled along two dimensions: the operator parallelism and the number of resources. In this paper, we show how these two dimensions, as two symbiotic entities, are independent but must mutually interact for the global benefit of the system. On the basis of this observation, we propose a fine-grained model for estimating the resource utilization of a stream processing application that enables the independent scaling of operators and resources. A simple, yet effective, combined management of the two dimensions allows us to propose ELYSIUM, a novel elastic scaling approach that provides efficient resource utilization. We implemented the proposed approach within Apache Storm and tested it by running two real-world applications with different input load curves. The outcomes backup our claims showing that the proposed symbiotic management outperforms elastic scaling strategies where operators and resources are jointly scaled.
2018
Cloud; elasticity; elastic scaling; stream processing; storm
01 Pubblicazione su rivista::01a Articolo in rivista
Elastic Symbiotic Scaling of Operators and Resources in Stream Processing Systems / Lombardi, Federico; Aniello, Leonardo; Bonomi, Silvia; Querzoni, Leonardo. - In: IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS. - ISSN 1045-9219. - STAMPA. - 29:3(2018), pp. 572-585. [10.1109/TPDS.2017.2762683]
File allegati a questo prodotto
File Dimensione Formato  
Lombardi_Postprint-Elastic-Symbolic-Scaling_2018.pdf

Open Access dal 19/10/2018

Note: 10.1109/TPDS.2017.2762683
Tipologia: Documento in Post-print (versione successiva alla peer review e accettata per la pubblicazione)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 3.14 MB
Formato Adobe PDF
3.14 MB Adobe PDF
Lombardi_Elastic-Symbolic-Scaling_2018.pdf

solo gestori archivio

Tipologia: Versione editoriale (versione pubblicata con il layout dell'editore)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 3.03 MB
Formato Adobe PDF
3.03 MB Adobe PDF   Contatta l'autore

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11573/1072968
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 44
  • ???jsp.display-item.citation.isi??? 32
social impact