Impact of Multimodal Emotional Input on User Experience in Chatbots

This study investigates whether incorporating multimodal emotional input (self-reported emotion and facial emotion) can enhance users’ emotional experiences in interactions with LLM-based chatbots. Twenty-eight participants engaged with four chatbot conditions: (A) text-only, (B) self-reported emotion, (C) facial emotion, and (D) combined input. Despite the hypothesis, perceived empathy was slightly higher in conditions without facial emotion input. However, participants with lower trait emotional expressivity reported more positive affect when facial emotion cues were included. Qualitative interviews further revealed varying perceptions of chatbot sensitivity to emotional needs. Participants with prior chatbot experience also felt a stronger social connection when conversations began with emotional topics. Additionally, older adults, women, and experienced users rated their interactions more positively. These findings suggest that emotional input may not universally enhance affective experiences but could benefit specific user profiles. The study advocates for a personalized, trait-sensitive approach to emotion-aware chatbot design rather than a one-size-fits-all multimodal strategy.