An enhanced speech emotion recognition using vision transformer

Abstract

In human-computer interaction systems, speech emotion recognition (SER) plays a crucial role because it enables computers to understand and react to users' emotions. In the past, SER has significantly emphasised acoustic properties extracted from speech signals. The use of visual signals for enhancing SER performance, however, has been made possible by recent developments in deep learning and computer vision. This work utilizes a lightweight Vision Transformer (ViT) model to propose a novel method for improving speech emotion recognition. We leverage the ViT model's capabilities to capture spatial dependencies and high-level features in images which are adequate indicators of emotional states from mel spectrogram input fed into the model. To determine the efficiency of our proposed approach, we conduct a comprehensive experiment on two benchmark speech emotion datasets, the Toronto English Speech Set (TESS) and the Berlin Emotional Database (EMODB). The results of our extensive experiment demonstrate a considerable improvement in speech emotion recognition accuracy attesting to its generalizability as it achieved 98%, 91%, and 93% (TESS-EMODB) accuracy respectively on the datasets. The outcomes of the comparative experiment show that the non-overlapping patch-based feature extraction method substantially improves the discipline of speech emotion recognition. Our research indicates the potential for integrating vision transformer models into SER systems, opening up fresh opportunities for real-world applications requiring accurate emotion recognition from speech compared with other state-of-the-art techniques.

Keywords: CNN; Deep learning; Human–computer interaction; Mel spectrogram; Speech emotion recognition; Vision transformer.

Figure 1

Traditional speech emotion recognition framework.

Figure 2

Propose Vision Transformer Architectural Framework.

Figure 3

Mel-spectrogram of selected emotion.

Figure 4

TESS dataset emotion distribution.

Figure 5

EMODB dataset emotion distribution.

Figure 6

The figure illustrates the proposed model’s performance loss curve for the three benchmarked datasets. (a) Loss diagram on TESS dataset (b) Loss diagram on EMODB dataset and (c) Loss diagram on TESS-EMODB dataset.

Figure 7

Summary of classification report for F1-Score, Recall and Precision.

Figure 8

Confusion matrix for TESS, EMODB and TESS-EMODB.

Figure 9

Test sample of emotion recognition output of the proposed model on three datasets: (i) represents recognition output on TESS dataset (ii) represents recognition output on EMODB dataset (iii) represent recognition output on TESS-EMODB dataset.