Multimodal Emotion Recognition using  Transformers and Cross-modal attention

Sarvesh Kamble; Khushi Pardeshi; Vaidehi Pate; Karishma Shah; Priyanka Deshpande

Multimodal Emotion Recognition using Transformers and Cross-modal attention

Authors: Sarvesh Kamble, Khushi Pardeshi, Vaidehi Pate, Karishma Shah, Priyanka Deshpande

Unique Paper ID: 182424
Volume: 12
Issue: 2
PageNo: 1910-1915

Keywords: Multimodal Feature Extraction Emotion Classification Temporal Emotion Dynamics Attention Mechanism Speech Emotion Recognition Facial Expression Analysis Cross- Modal Transformer Audio-Visual Synchronization Deep Neural Networks Auto-Encoders Emotion Intensity Prediction Valence- Arousal Modeling Real-Time Emotion Detection Human- Computer Interaction.

Abstract:
Technologies under affective computing require multimodal emotion recognition through audio-visual information on system interfaces to detect emotions. Transformer and Cross-Modal Attention present a new architecture designed for Multimodal Emotion Recognition which facilitates semantic pattern connecting and temporal pattern retrieval between face and voice signals. The system employs RAVDESS and FER+ datasets for training and evaluation purposes to evaluate emotional states in various conditions. The system achieves long-term dependencies in each individual information stream using transformer encoders while connecting important features between audio and visual sections through cross-modal attention. Emotion classification methods require multiple set of modal representation data to be merged into unified representation packages by using fusion algorithms. Multimodal learning with attention-based emotion detection achieves superior performance than single-mode benchmarks according to the design structure. The system overcome temporal mismatches together with inconsistent attributes between different modalities using attention- guided refinement in joint optimization procedures. The real-time recognition system provides practical solutions through systematic methods that prove useful for human-machine interaction control as well as healthcare surveillance and health monitoring.

Download article

email to a friend

Cite This Article

ISSN: 2349-6002
Volume: 12
Issue: 2
PageNo: 1910-1915

Multimodal Emotion Recognition using Transformers and Cross-modal attention

Available:https://ijirt.org/Article?manuscript=182424

Impact Factor
8.01 (Year 2024)

UGC Approved
Journal no 47859

Join Our IPN

IJIRT Partner Network

Submit your research paper and those of your network (friends, colleagues, or peers) through your IPN account, and receive 800 INR for each paper that gets published.

Join Now