Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Jun 1:14:1141801.
doi: 10.3389/fpsyg.2023.1141801. eCollection 2023.

Multi-Input CNN-LSTM deep learning model for fear level classification based on EEG and peripheral physiological signals

Nagisa Masuda  1 , Ikuko Eguchi Yairi  1
Affiliations

Multi-Input CNN-LSTM deep learning model for fear level classification based on EEG and peripheral physiological signals

Nagisa Masuda et al. Front Psychol. .

Abstract

Objective and accurate classification of fear levels is a socially important task that contributes to developing treatments for Anxiety Disorder, Obsessive-compulsive Disorder, Post-Traumatic Stress Disorder (PTSD), and Phobia. This study examines a deep learning model to automatically estimate human fear levels with high accuracy using multichannel EEG signals and multimodal peripheral physiological signals in the DEAP dataset. The Multi-Input CNN-LSTM classification model combining Convolutional Neural Network (CNN) and Long Sort-Term Memory (LSTM) estimated four fear levels with an accuracy of 98.79% and an F1 score of 99.01% in a 10-fold cross-validation. This study contributes to the following; (1) to present the possibility of recognizing fear emotion with high accuracy using a deep learning model from physiological signals without arbitrary feature extraction or feature selection, (2) to investigate effective deep learning model structures for high-accuracy fear recognition and to propose Multi-Input CNN-LSTM, and (3) to examine the model's tolerance to individual differences in physiological signals and the possibility of improving accuracy through additional learning.

Keywords: Convolutional Neural Network; EEG; emotion recognition; fear; long-term and short-term memory; physiological signal.

PubMed Disclaimer

Conflict of interest statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Figures

Figure 1
Figure 1
The flowchart of the proposed method.
Figure 2
Figure 2
The network architecture of the Multi-Input CNN-LSTM model proposed in this paper. BN is the Batch Normalization, DO is the Dropout, and FC is the Fully Connected.
Figure 3
Figure 3
Shows the box plots of the dropout rate of the Dropout layers. 1st and 2nd Drop layers are DO1 and DO2 in Figure 2. The horizontal axis is the outer hold, and the vertical axis is the dropout rate. The values of five inner holds for each outer hold are plotted in each box.
Figure 4
Figure 4
Shows the box plots of the learning rate. The horizontal axis is the outer hold, and the vertical axis is the learning rate. The values of five inner holds for each outer hold are plotted in each box.
Figure 5
Figure 5
The Receiver Operating Characteristic (ROC) Curve and Area Under the Curve (AUC) values of the Multi-Input CNN-LSTM model proposed in this paper.
Figure 6
Figure 6
The confusion matrix of the four evaluation methods. The vertical axis is the true label, and the horizontal axis is the prediction label. The diagonal values represent the recall for each class, and the sum of the row is 100%. The contents of the parentheses indicate the number of data.
See this image and copyright information in PMC

References

    1. Abadi M. K., Subramanian R., Kia S. M., Avesani P., Patras I., Sebe N. (2015). DECAF: MEG-based multimodal database for decoding affective physiological responses. IEEE Trans. Affect. Comput. 6, 209–222. doi: 10.1109/TAFFC.2015.2392932 - DOI
    1. Ahmed M. Z. I., Sinha N., Ghaderpour E., Phadikar S., Ghosh R. (2023). A novel baseline removal paradigm for subject-independent features in emotion classification using EEG. Bioengineering 10:54. doi: 10.3390/bioengineering10010054 - DOI - PMC - PubMed
    1. Akiba T., Sano S., Yanase T., Ohta T., Koyama M. (2019). “Optuna: a next-generation hyperparameter optimization framework” in Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery and data mining (KDD) (New York: Association for Computing Machinery; ), 2623–2631.
    1. Alhagry S., Fahmy A. A., El-Khoribi R. A. (2017). Emotion recognition based on EEG using LSTM recurrent neural network. Int. J. Adv. Comput. Sci. Appl. 8:10. doi: 10.14569/IJACSA.2017.081046 - DOI
    1. Altman N. S. (1992). An introduction to kernel and nearest-neighbor nonparametric regression. Am. Stat. 46, 175–185. doi: 10.2307/2685209 - DOI

LinkOut - more resources