--- id: affective-computing title: "Affective Computing: Multimodal Emotion Recognition, Sentiment Analysis, and Empathetic AI" schema_type: article category: ai language: en confidence: high last_verified: "2026-05-24" created_date: "2026-05-24" generation_method: ai_assisted ai_models: - claude-4.5-sonnet derived_from_human_seed: true conflict_of_interest: none_declared is_live_document: false data_period: static completeness: 0.85 atomic_facts: - id: af-affective-computing-1 statement: >- IEEE (November 2025) published a comprehensive survey of multimodal emotion recognition covering four modalities — speech, facial expressions, physiological signals (EEG, ECG), and text — documenting that transformer-based cross-modal fusion architectures achieve 8-15% higher accuracy than unimodal baselines, with the MemoCMT model (Nature 2025) demonstrating cross-modal transformer attention to resolve modality conflicts (e.g., smiling face + angry voice). source_title: "IEEE (2025) — Multimodal Emotion Recognition: A Comprehensive Survey / Nature (2025) — MemoCMT: Cross-modal transformer — doi:10.1038/s41598-025-89202-x" source_url: https://ieeexplore.ieee.org/document/11264591 confidence: high - id: af-affective-computing-2 statement: >- ScienceDirect (October 2025) introduced EmoVerse — a framework enhancing multimodal LLMs for affective computing with a multitask dataset (AMT) supporting sentiment analysis, emotion recognition, and emotion cause detection — demonstrating that LLMs fine-tuned with affective reasoning prompts achieve human-level performance on dimensional emotion prediction (valence, arousal, dominance) and outperform task-specific models by 6-10% on emotion cause identification. source_title: "ScienceDirect Neurocomputing (2025) — EmoVerse: Enhancing MLLMs for Affective Computing — doi:10.1016/j.neucom.2025.129953" source_url: https://www.nature.com/articles/s41598-025-89202-x confidence: high primary_sources: - id: ps-affective-computing-1 title: "Multimodal Emotion Recognition: A Comprehensive Survey of Methods, Modalities, and Applications" type: academic_paper year: 2025 institution: IEEE Access / ACM url: https://ieeexplore.ieee.org/document/11264591 - id: ps-affective-computing-2 title: "MemoCMT: multimodal emotion recognition using cross-modal transformer" type: academic_paper year: 2025 institution: Nature Scientific Reports doi: 10.1038/s41598-025-89202-x url: https://www.nature.com/articles/s41598-025-89202-x known_gaps: - Cultural variation in emotional expression — models trained on Western data fail on non-Western cultures - Continuous emotion tracking over long conversations (not just per-utterance classification) disputed_statements: [] secondary_sources: - title: "Affective Computing: A Comprehensive Survey of Emotion Recognition, Sentiment Analysis, and Affective AI" type: survey_paper year: 2024 authors: - multiple institution: IEEE TPAMI url: https://doi.org/10.1109/TPAMI.2024.3385267 - title: Affective Computing (Book — Picard) type: textbook year: 1997 authors: - Picard, Rosalind W. institution: MIT Press url: https://mitpress.mit.edu/9780262661157/ - title: "Deep Learning for Emotion Recognition: A Comprehensive Survey of Multimodal Affective Computing" type: survey_paper year: 2025 authors: - multiple institution: Information Fusion (Elsevier) url: https://doi.org/10.1016/j.inffus.2025.102890 - title: "The Handbook of Multimodal-Multisensor Interfaces: Emotion Recognition, Affective Computing" type: textbook year: 2019 authors: - Oviatt, Sharon - Schuller, Björn - Cohen, Philip - et al. institution: ACM Books / Morgan & Claypool url: https://doi.org/10.1145/3233795 updated: "2026-05-24" --- ## TL;DR Affective computing gives AI emotional intelligence — recognizing human emotions from voice, face, text, and physiology, and responding empathetically. From mental health monitoring to customer service and autonomous driving (detecting driver stress), emotion-aware AI is transitioning from academic research to production deployment. ## Core Explanation Emotion models: (1) Categorical — emotions are discrete (Ekman's six: happiness, sadness, anger, fear, disgust, surprise) plus neutral. Used in most classification benchmarks; (2) Dimensional — emotions vary continuously along valence (positive-negative), arousal (calm-excited), and dominance (controlled-submissive). Captures nuanced states (nostalgia, frustration) that categorical models miss. Modalities: (A) Facial expression — CNNs/Vision Transformers process face images/video, detecting Action Units (AU) from Facial Action Coding System (FACS). Landmark detection → expression classification; (B) Speech — prosody (pitch, energy, rate), spectral features (MFCCs, spectrograms) processed by CNNs/LSTMs/Transformers. Cross-lingual emotion in speech is particularly challenging; (C) Text — sentiment analysis via fine-tuned transformers (BERT, RoBERTa, emotion-specific models); (D) Physiological — EEG (brain), ECG (heart rate variability), GSR (skin conductance), providing ground-truth emotional signals not subject to social masking. ## Detailed Analysis Multimodal fusion strategies: (E) Early fusion — concatenate all modality features before classification; (L) Late fusion — classify each modality independently, ensemble predictions; (H) Hybrid/cross-modal — attention mechanisms learn which modality to trust when modalities conflict. MemoCMT (Nature 2025): cross-modal transformer processes speech and facial features simultaneously, learning to attend to the facial stream when speech is ambiguous and vice versa. Achieves 82% accuracy on IEMOCAP (4-class emotion), improving 8% over best unimodal. EmoVerse (ScienceDirect 2025): extends multimodal LLMs (LLaVA, GPT-4V) with affective reasoning — the model generates not just emotion labels but explanations ("The person appears sad because their speech rate slowed and they mentioned loss"). Applications: (1) Mental health — detecting depression/anxiety from speech and text patterns; (2) Education — detecting confusion and engagement in online learning; (3) Automotive — driver emotion/stress monitoring for safety; (4) Customer service — real-time agent coaching based on customer emotion; (5) Social robotics — empathetic response generation. Key challenge: cultural variability — a smile means different things across cultures; training data is overwhelmingly Western (IEMOCAP, RAVDESS are English-only). The 2025 IEEE survey calls for culturally-diverse benchmarks. ## Further Reading - IEMOCAP, RAVDESS, MELD Emotion Datasets - Dimensional Emotion Model (Russell's Circumplex) - OpenFace: Facial Action Unit Detection