Multiscale graph and multimodal data fusion for ECG emotion detection

Affective computing research has boosted the interest in identifying mental states through physiological signals obtained from wearable devices. Increasingly sophisticated and accessible wearable electrocardiogram (ECG) devices provide real-time data that can facilitate insights into an individual’s emotional well-being. Herein we propose novel graph-based multiscale ECG features and apply them for emotion classification. Specifically, we analyze the persistence of energy distribution of an ECG signal at different wavelet scales and we represent it by a graph, where higher edge weights represent higher similarity. In this scenario, each graph node represents one wavelet scale and the link between them is measured as the correlation coefficient between the autocorrelation functions of the wavelets at the two nodes. The graph structure can be adopted as a feature for emotion classification. Then, we extend the approach to multimodal measurements, where the info conveyed by different physiological signal is encompassed by a multilayer graph. We test the ability of our approach to identify human emotional states measured with ECG and GRS measurements. Our numerical results show that the proposed graph-based features are able to accurately capture emotion and their changes. As a side advantage, the proposed representation of multimodal physiological signals in a graph domain is intrinsically robust to biometric information leakage.