Speech emotion classification using attention based network and regularized feature selection

Abstract

Speech emotion classification (SEC) has gained the utmost height and occupied a conspicuous position within the research community in recent times. Its vital role in Human-Computer Interaction (HCI) and affective computing cannot be overemphasized. Many primitive algorithmic solutions and deep neural network (DNN) models have been proposed for efficient recognition of emotion from speech however, the suitability of these methods to accurately classify emotion from speech with multi-lingual background and other factors that impede efficient classification of emotion is still demanding critical consideration. This study proposed an attention-based network with a pre-trained convolutional neural network and regularized neighbourhood component analysis (RNCA) feature selection techniques for improved classification of speech emotion. The attention model has proven to be successful in many sequence-based and time-series tasks. An extensive experiment was carried out using three major classifiers (SVM, MLP and Random Forest) on a publicly available TESS (Toronto English Speech Sentence) dataset. The result of our proposed model (Attention-based DCNN+RNCA+RF) achieved 97.8% classification accuracy and yielded a 3.27% improved performance, which outperforms state-of-the-art SEC approaches. Our model evaluation revealed the consistency of attention mechanism and feature selection with human behavioural patterns in classifying emotion from auditory speech.

Figure 1

Conventional speech emotion classification system.

Figure 2

Structure of mel-spectrogram extraction.

Figure 3

Proposed model architecture.

Figure 4

Convolutional layers block diagram.

Figure 5

RNCA framework.

Figure 6

Attention-based Vgg16+RNCA+RF.

Figure 7

Attention-based Vgg16+RNCA+MLP.

Figure 8

Attention-based Vgg16+RNCA+SVM.

Figure 9

Attention-based Vgg19+RNCA+RF.

Figure 10

Attention-based Vgg19+RNCA+MLP.

Figure 11

Attention-based Vgg19+RNCA+SVM.

Figure 12

Performance chart with 4 metrics and 3 classifiers.

Figure 13

Model loss curve.

Figure 14

ROC curve.