Emotion recognition is relevant in various domains, ranging from healthcare to human-computer interaction. Physiological signals, being beyond voluntary control, offer reliable information for this purpose, unlike speech and facial expressions which can be controlled at will. They reflect genuine emotional responses, devoid of conscious manipulation, thereby enhancing the credibility of emotion recognition systems. Nonetheless, multimodal emotion recognition with deep learning models remains a relatively unexplored field. In this paper, we introduce a fully hypercomplex network with a hierarchical learning structure to fully capture correlations. Specifically, at the encoder level, the model learns intra- modal relations among the different channels of each input signal. Then, a hypercomplex fusion module learns inter-modal relations among the embeddings of the different modalities. The main novelty is in exploiting intra-modal relations by endowing the encoders with parameterized hyper-complex convolutions (PHCs) that thanks to hypercomplex algebra can capture inter-channel interactions within single modalities. Instead, the fusion module comprises parameterized hypercomplex multiplications (PHMs) that can model inter-modal correlations. The proposed architecture surpasses state-of-the-art models on the MAHNOB-HCI dataset for emotion recognition, specifically in classifying valence and arousal from electroencephalograms (EEGs) and peripheral physiological signals. The code of this study is available at https://github.com/ispamm/MHyEEG.

Hierarchical hypercomplex network for multimodal emotion recognition / Lopez, Eleonora; Uncini, Aurelio; Comminiello, Danilo. - (2024), pp. 1-6. (Intervento presentato al convegno 2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP) tenutosi a London, UK) [10.1109/mlsp58920.2024.10734815].

Hierarchical hypercomplex network for multimodal emotion recognition

Lopez, Eleonora;Uncini, Aurelio;Comminiello, Danilo
2024

Abstract

Emotion recognition is relevant in various domains, ranging from healthcare to human-computer interaction. Physiological signals, being beyond voluntary control, offer reliable information for this purpose, unlike speech and facial expressions which can be controlled at will. They reflect genuine emotional responses, devoid of conscious manipulation, thereby enhancing the credibility of emotion recognition systems. Nonetheless, multimodal emotion recognition with deep learning models remains a relatively unexplored field. In this paper, we introduce a fully hypercomplex network with a hierarchical learning structure to fully capture correlations. Specifically, at the encoder level, the model learns intra- modal relations among the different channels of each input signal. Then, a hypercomplex fusion module learns inter-modal relations among the embeddings of the different modalities. The main novelty is in exploiting intra-modal relations by endowing the encoders with parameterized hyper-complex convolutions (PHCs) that thanks to hypercomplex algebra can capture inter-channel interactions within single modalities. Instead, the fusion module comprises parameterized hypercomplex multiplications (PHMs) that can model inter-modal correlations. The proposed architecture surpasses state-of-the-art models on the MAHNOB-HCI dataset for emotion recognition, specifically in classifying valence and arousal from electroencephalograms (EEGs) and peripheral physiological signals. The code of this study is available at https://github.com/ispamm/MHyEEG.
2024
2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP)
hypercomplex networks; multimodal emotion recognition; EEG; physiological signals
04 Pubblicazione in atti di convegno::04b Atto di convegno in volume
Hierarchical hypercomplex network for multimodal emotion recognition / Lopez, Eleonora; Uncini, Aurelio; Comminiello, Danilo. - (2024), pp. 1-6. (Intervento presentato al convegno 2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP) tenutosi a London, UK) [10.1109/mlsp58920.2024.10734815].
File allegati a questo prodotto
File Dimensione Formato  
Lopez_Hierarchical_2024.pdf

solo gestori archivio

Tipologia: Versione editoriale (versione pubblicata con il layout dell'editore)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 403.83 kB
Formato Adobe PDF
403.83 kB Adobe PDF   Contatta l'autore
Lopez_Frontespizio_Hierarchical_2024.pdf

solo gestori archivio

Note: Frontespizio
Tipologia: Altro materiale allegato
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 373.98 kB
Formato Adobe PDF
373.98 kB Adobe PDF   Contatta l'autore
Lopez_Indice_Hierarchical_2024.pdf

solo gestori archivio

Note: Indice
Tipologia: Altro materiale allegato
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 6.92 MB
Formato Adobe PDF
6.92 MB Adobe PDF   Contatta l'autore

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11573/1725066
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact