Several cybersecurity domains, such as ransomware detection, forensics and data analysis, require methods to reliably identify encrypted data fragments. Typically, current approaches employ statistics derived from byte-level distribution, such as entropy estimation, to identify encrypted fragments. However, modern content types use compression techniques which alter data distribution pushing it closer to the uniform distribution. The result is that current approaches exhibit unreliable encryption detection performance when compressed data appear in the dataset. Furthermore, proposed approaches are typically evaluated over few data types and fragment sizes, making it hard to assess their practical applicability. This paper compares existing statistical tests on a large, standardized dataset and shows that current approaches consistently fail to distinguish encrypted and compressed data on both small and large fragment sizes. We address these shortcomings and design ENCOD, a learning-based classifier which can reliably distinguish compressed and encrypted data. We evaluate ENCOD on a dataset of 16 different file types and fragment sizes ranging from 512B to 8KB. Our results highlight that ENCOD outperforms current approaches by a wide margin, with accuracy ranging from ∼82% for 512B fragments up to ∼92% for 8KB data fragments. Moreover, ENCOD can pinpoint the exact format of a given data fragment, rather than performing only binary classification like previous approaches.

Reliable detection of compressed and encrypted data / De Gaspari, F.; Hitaj, D.; Pagnotta, G.; De Carli, L.; Mancini, L. V.. - In: NEURAL COMPUTING & APPLICATIONS. - ISSN 1433-3058. - 34:22(2022), pp. 20379-20393. [10.1007/s00521-022-07586-7]

Reliable detection of compressed and encrypted data

De Gaspari F.
Membro del Collaboration Group
;
Hitaj D.
Membro del Collaboration Group
;
Pagnotta G.
Membro del Collaboration Group
;
De Carli L.
Membro del Collaboration Group
;
Mancini L. V.
Membro del Collaboration Group
2022

Abstract

Several cybersecurity domains, such as ransomware detection, forensics and data analysis, require methods to reliably identify encrypted data fragments. Typically, current approaches employ statistics derived from byte-level distribution, such as entropy estimation, to identify encrypted fragments. However, modern content types use compression techniques which alter data distribution pushing it closer to the uniform distribution. The result is that current approaches exhibit unreliable encryption detection performance when compressed data appear in the dataset. Furthermore, proposed approaches are typically evaluated over few data types and fragment sizes, making it hard to assess their practical applicability. This paper compares existing statistical tests on a large, standardized dataset and shows that current approaches consistently fail to distinguish encrypted and compressed data on both small and large fragment sizes. We address these shortcomings and design ENCOD, a learning-based classifier which can reliably distinguish compressed and encrypted data. We evaluate ENCOD on a dataset of 16 different file types and fragment sizes ranging from 512B to 8KB. Our results highlight that ENCOD outperforms current approaches by a wide margin, with accuracy ranging from ∼82% for 512B fragments up to ∼92% for 8KB data fragments. Moreover, ENCOD can pinpoint the exact format of a given data fragment, rather than performing only binary classification like previous approaches.
2022
encryption detection; forensics; machine learning; traffic analysis
01 Pubblicazione su rivista::01a Articolo in rivista
Reliable detection of compressed and encrypted data / De Gaspari, F.; Hitaj, D.; Pagnotta, G.; De Carli, L.; Mancini, L. V.. - In: NEURAL COMPUTING & APPLICATIONS. - ISSN 1433-3058. - 34:22(2022), pp. 20379-20393. [10.1007/s00521-022-07586-7]
File allegati a questo prodotto
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11573/1658422
 Attenzione

Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo

Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 13
  • ???jsp.display-item.citation.isi??? 4
social impact