Modern parallel/distributed simulations can produce large amounts of data. The historical approach of performing analyses at the end of the simulation is unlikely to cope with modern, extremely large-scale analytics jobs. Indeed, the I/O subsystem can quickly become the global bottleneck. Similarly, processing onthe-fly the data produced by simulations can significantly impair the performance in terms of computational capacity and network load. We present a methodology and reference architecture for constructing an autonomic control system to determine at runtime the best placement for data processing (on simulation nodes or a set of external nodes). This allows for a good tradeoff between the load on the simulation’s critical path and the data communication system. Our preliminary experimentation shows that autonomic orchestration is crucial to improve the global performance of a data analysis system, especially when the simulation node’s rate of data production varies during simulation.

Autonomic Orchestration Of In-Situ and In-Transit Data Analytics For Simulation Studies / Du, Xiaorui; Pimpini, Adriano; Piccione, Andrea; Meng, Zhioxiao; Siguenza-Torres, Anibal; Bortoli, Stefano; Knoll, Alois; Pellegrini, Alessandro. - (2023). (Intervento presentato al convegno 2023 Winter Simulation Conference tenutosi a San Antonio, Texas, USA).

Autonomic Orchestration Of In-Situ and In-Transit Data Analytics For Simulation Studies

Adriano Pimpini
Secondo
;
Andrea Piccione;Alessandro Pellegrini
Ultimo
2023

Abstract

Modern parallel/distributed simulations can produce large amounts of data. The historical approach of performing analyses at the end of the simulation is unlikely to cope with modern, extremely large-scale analytics jobs. Indeed, the I/O subsystem can quickly become the global bottleneck. Similarly, processing onthe-fly the data produced by simulations can significantly impair the performance in terms of computational capacity and network load. We present a methodology and reference architecture for constructing an autonomic control system to determine at runtime the best placement for data processing (on simulation nodes or a set of external nodes). This allows for a good tradeoff between the load on the simulation’s critical path and the data communication system. Our preliminary experimentation shows that autonomic orchestration is crucial to improve the global performance of a data analysis system, especially when the simulation node’s rate of data production varies during simulation.
2023
2023 Winter Simulation Conference
Simulation; Data Analytics; In-transit processing
04 Pubblicazione in atti di convegno::04b Atto di convegno in volume
Autonomic Orchestration Of In-Situ and In-Transit Data Analytics For Simulation Studies / Du, Xiaorui; Pimpini, Adriano; Piccione, Andrea; Meng, Zhioxiao; Siguenza-Torres, Anibal; Bortoli, Stefano; Knoll, Alois; Pellegrini, Alessandro. - (2023). (Intervento presentato al convegno 2023 Winter Simulation Conference tenutosi a San Antonio, Texas, USA).
File allegati a questo prodotto
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11573/1697986
 Attenzione

Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo

Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact