Effect on speech emotion classification of a feature selection approach using a convolutional neural network

Abstract

Speech emotion recognition (SER) is a challenging issue because it is not clear which features are effective for classification. Emotionally related features are always extracted from speech signals for emotional classification. Handcrafted features are mainly used for emotional identification from audio signals. However, these features are not sufficient to correctly identify the emotional state of the speaker. The advantages of a deep convolutional neural network (DCNN) are investigated in the proposed work. A pretrained framework is used to extract the features from speech emotion databases. In this work, we adopt the feature selection (FS) approach to find the discriminative and most important features for SER. Many algorithms are used for the emotion classification problem. We use the random forest (RF), decision tree (DT), support vector machine (SVM), multilayer perceptron classifier (MLP), and k-nearest neighbors (KNN) to classify seven emotions. All experiments are performed by utilizing four different publicly accessible databases. Our method obtains accuracies of 92.02%, 88.77%, 93.61%, and 77.23% for Emo-DB, SAVEE, RAVDESS, and IEMOCAP, respectively, for speaker-dependent (SD) recognition with the feature selection method. Furthermore, compared to current handcrafted feature-based SER methods, the proposed method shows the best results for speaker-independent SER. For EMO-DB, all classifiers attain an accuracy of more than 80% with or without the feature selection technique.

Keywords: Convolutional neural network; Data augmentation; Feature extraction; Feature selection; Mel-spectrogram; Speech emotion recognition.

Figure 1. The structure of our proposed model for audio emotion recognition.

Figure 2. The general architecture of AlexNet, The parameters of the convolutional layer are represented by the “Conv(kernel size)-[stride size]-[number of channels]”.

The parameters of themax-pooling layer are indicated as “Maxpool-[kernel size]-[stride size]”.

Figure 3. Confusion matrix obtained by the SVM on the Emo-DB database for the SD experiment.

Figure 4. Confusion matrix obtained by the SVM on the SAVEE database for the SD experiment.

Figure 5. Confusion matrix obtained by the SVM on the RAVDESS database for the SD experiment.

Figure 6. Confusion matrix obtained by the MLP on the IEMOCAP database for the SD experiment.

Figure 7. Confusion matrix obtained by the SVM on the RAVDESS database for the SI experiment.

Figure 8. Confusion matrix obtained by the MLP on the RAVDESS database for the SI experiment.

Figure 9. Confusion matrix obtained by the SVM on the IEMOCAP database for the SI experiment.