In recent years, hyper-complex deep networks (such as complex-valued and quaternion-valued neural networks - QVNNs) have received a renewed interest in the literature. They find applications in multiple fields, ranging from image reconstruction to 3D audio processing. Similar to their real-valued counterparts, quaternion neural networks require custom regularisation strategies to avoid overfitting. In addition, for many real-world applications and embedded implementations, there is the need of designing sufficiently compact networks, with few weights and neurons. However, the problem of regularising and/or sparsifying QVNNs has not been properly addressed in the literature as of now. In this study, the authors show how to address both problems by designing targeted regularisation strategies, which can minimise the number of connections and neurons of the network during training. To this end, they investigate two extensions of l1and structured regularisations to the quaternion domain. In the authors' experimental evaluation, they show that these tailored strategies significantly outperform classical (realvalued) regularisation approaches, resulting in small networks especially suitable for low-power and real-time applications.

Compressing deep-quaternion neural networks with targeted regularisation / Vecchi, R.; Scardapane, S.; Comminiello, D.; Uncini, A.. - In: CAAI TRANSACTIONS ON INTELLIGENCE TECHNOLOGY. - ISSN 2468-6557. - 5:3(2020), pp. 172-176. [10.1049/trit.2020.0020]

Compressing deep-quaternion neural networks with targeted regularisation

Scardapane S.;Comminiello D.;Uncini A.
2020

Abstract

In recent years, hyper-complex deep networks (such as complex-valued and quaternion-valued neural networks - QVNNs) have received a renewed interest in the literature. They find applications in multiple fields, ranging from image reconstruction to 3D audio processing. Similar to their real-valued counterparts, quaternion neural networks require custom regularisation strategies to avoid overfitting. In addition, for many real-world applications and embedded implementations, there is the need of designing sufficiently compact networks, with few weights and neurons. However, the problem of regularising and/or sparsifying QVNNs has not been properly addressed in the literature as of now. In this study, the authors show how to address both problems by designing targeted regularisation strategies, which can minimise the number of connections and neurons of the network during training. To this end, they investigate two extensions of l1and structured regularisations to the quaternion domain. In the authors' experimental evaluation, they show that these tailored strategies significantly outperform classical (realvalued) regularisation approaches, resulting in small networks especially suitable for low-power and real-time applications.
2020
quaternion; neural network; compression; regularization
01 Pubblicazione su rivista::01a Articolo in rivista
Compressing deep-quaternion neural networks with targeted regularisation / Vecchi, R.; Scardapane, S.; Comminiello, D.; Uncini, A.. - In: CAAI TRANSACTIONS ON INTELLIGENCE TECHNOLOGY. - ISSN 2468-6557. - 5:3(2020), pp. 172-176. [10.1049/trit.2020.0020]
File allegati a questo prodotto
File Dimensione Formato  
Vecchi_Compressing_2020.pdf

accesso aperto

Tipologia: Versione editoriale (versione pubblicata con il layout dell'editore)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 502.29 kB
Formato Adobe PDF
502.29 kB Adobe PDF

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11573/1503651
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 13
  • ???jsp.display-item.citation.isi??? 10
social impact