Multimodal Assessment of Mental States and Visual Search for a User-Centred Design of Semantic Web Platforms

Featured Application: The proposed multimodal framework can be applied to the objective evaluation and optimization of e-learning authoring platforms, supporting the identification of usability bottlenecks that are not detectable through traditional questionnaires alone. By combining neurophysiological, eye tracking, and interaction-based metrics, the approach enables designers and developers to perform data-driven, user-centred improvements of complex web interfaces. This methodology can be adopted in iterative UX design processes to enhance platform usability, reduce cognitive load during content creation, and ultimately facilitate the adoption of digital educational technologies by instructors and non-expert users. Background: Digital learning platforms increasingly leverage semantic web technologies to support interoperable and adaptive e-learning. However, the usability and cognitive impact of web-based authoring tools are still mainly assessed through subjective questionnaires and interaction logs, which provide limited time resolution and weak diagnostic power for identifying specific interface bottlenecks. Methods: We propose a multimodal evaluation of SOULSS, a semantic web-oriented platform for creating and optimizing digital learning contents. Eighteen participants completed an authoring workflow organized into three macro-segments (tutorial, initialization, module creation) while wearable electroencephalography, electrodermal activity, photoplethysmography, and eye tracking were recorded; objective metrics were analyzed both across macro-segments and within predefined micro-activities, whereas subjective engagement was collected after each macro-segment using the UES-SF. Results: Objective measures indicated increased EEG-derived mental workload and stress, higher tonic sympathetic arousal, and greater visual search and interaction effort during initialization and module creation, while UES-SF scores were lower during initialization. Fine-grained analyses localized critical elements to tutorial navigation options, the new course entry point, and spoiler-related controls. Repeated-measures correlations linked subjective scores with objective markers and supported an association between stress-related activation and delayed visual discovery. Conclusions: Integrating neurophysiological and eye tracking measures enables a more diagnostic assessment of semantic web-based authoring platforms than questionnaires alone, providing actionable evidence for iterative UX optimization and supporting a more user-centred design of digital educational tools.