A Review of Emotion Recognition Using Physiological Signals

Abstract

Emotion recognition based on physiological signals has been a hot topic and applied in many areas such as safe driving, health care and social security. In this paper, we present a comprehensive review on physiological signal-based emotion recognition, including emotion models, emotion elicitation methods, the published emotional physiological datasets, features, classifiers, and the whole framework for emotion recognition based on the physiological signals. A summary and comparation among the recent studies has been conducted, which reveals the current existing problems and the future work has been discussed.

Keywords: classifiers; emotion model; emotion recognition; emotion stimulation; features; physiological signals.

Figure 1

Plutchik’s Wheel of Emotions.

Figure 2

2D emotion space model.

Figure 3

3D emotion space model.

Figure 4

(a) MAUI—Multimodal Affective User Interface; (b) Framework of AVRS; (c) VR scenes cut show.

Figure 5

Position of the bio-sensors.

Figure 6

(a) Mean accuracy of different channels; (b) The performance of different windows sizes; (c) The average accuracies of GELM; (d) Spectrogram shows different patterns with different emotions.

Figure 7

(a) Logical scheme of the overall short-time emotion recognition concept; (b) Instantaneous tracking of the HR V indices computed from a representative subject using the proposed NARI model during the passive emotional elicitation (two neutral sessions alternated to a L-M and a M-H arousal session); (c) Diagram of the proposed method; (d) Experimental results.

Figure 8

(a) The Emotion Check device; (b) Diagram describing the components of the Emotion Check device; (c) Prototype of glove with sensor unit; (d) Body Media Sense Wear Armband; (e) Left: The physiological measures of EMG and EDA. Middle: The physiological measures of EEG, BVP and TMP. Right: The physiological measures of physiological sensors in the experiments; (g) Illustration of R-TIPS. This platform allows wireless monitoring of cardiac signals. It consists of a transmitter system and three sensors; (f) The transmitter system is placed on the participant’s hip, and the sensors are placed below right breast, on the right side, and on the back.

Figure 9

(a) Monitoring of epileptic seizures using EDA; (b,c) Wearable GSR sensor.

Figure 10

Emotion recognition process using physiological signals under target emotion stimulation.

Figure 11

(a) The decomposition of R-R interval signal (emotion of sadness); (b) The structure of Autoencoder; (c) The structure of Bimodal Deep AutoEncoder.

Figure 12

(a) Typical framework of multimodal information fusion; (b) SVM results for different emotions with EEG frequency band; (c) Demo of the proposed feature level fusion. A feature vector created at any time step is valid for the next two steps.

Figure 13

Classification models.

Figure 14

(a) The structure of standard RNN and LSTM; (b) The structure and settings of CNN.

Figure 15

ROC.

Figure 16

Comparative results on the same public accessible datasets: (a) DEAP; (b) MAHNOB; database; (c) SEED.

Figure 17

The comparation of recognition rate among previous research.

Figure 18

Subject-dependent and Subject-independent recognition rate. (The horizontal axis represents the same sequence number as the Table 3. Different colors represent different classification categories: Blue—two categories, yellow—three categories, green four—categories, grey—five categories, purple—six categories).