Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023;15(4):1819-1830.
doi: 10.1007/s41870-023-01183-0. Epub 2023 Mar 21.

Facial expression recognition in videos using hybrid CNN & ConvLSTM

Rajesh Singh  1 , Sumeet Saurav  2 , Tarun Kumar  3 , Ravi Saini  2 , Anil Vohra  1 , Sanjay Singh  2
Affiliations

Facial expression recognition in videos using hybrid CNN & ConvLSTM

Rajesh Singh et al. Int J Inf Technol. 2023.

Abstract

The three-dimensional convolutional neural network (3D-CNN) and long short-term memory (LSTM) have consistently outperformed many approaches in video-based facial expression recognition (VFER). The image is unrolled to a one-dimensional vector by the vanilla version of the fully-connected LSTM (FC-LSTM), which leads to the loss of crucial spatial information. Convolutional LSTM (ConvLSTM) overcomes this limitation by performing LSTM operations in convolutions without unrolling, thus retaining useful spatial information. Motivated by this, in this paper, we propose a neural network architecture that consists of a blend of 3D-CNN and ConvLSTM for VFER. The proposed hybrid architecture captures spatiotemporal information from the video sequences of emotions and attains competitive accuracy on three FER datasets open to the public, namely the SAVEE, CK + , and AFEW. The experimental results demonstrate excellent performance without external emotional data with the added advantage of having a simple model with fewer parameters. Moreover, unlike the state-of-the-art deep learning models, our designed FER pipeline improves execution speed by many factors while achieving competitive recognition accuracy. Hence, the proposed FER pipeline is an appropriate candidate for recognizing facial expressions on resource-limited embedded platforms for real-time applications.

Keywords: 3D convolutional neural networks (3D-CNN); Convolutional LSTM (ConvLSTM); Long short-term memory (LSTM); Video-based facial expression recognition (VFER).

PubMed Disclaimer

Figures

Fig. 1
Fig. 1
Proposed framework for video-based facial expression recognition
Fig. 2
Fig. 2
Prototypical facial image with 68 landmarks
Fig. 3
Fig. 3
Illustration of 3D convolution operation
Fig. 4
Fig. 4
Schematic representation of the proposed hybrid 3D-CNN & ConvLSTM model
Fig. 5
Fig. 5
Internal elements of the ConvLSTM
Fig. 6
Fig. 6
A sample facial expression sequence from the CK + dataset (top to bottom and left to right): Displays evolution of the happiness expression
Fig. 7
Fig. 7
Confusion matrix on the CK + database
Fig. 8
Fig. 8
Confusion Matrix on the SAVEE database
Fig. 9
Fig. 9
Confusion Matrix on the randomly selected 20% of AFEW samples
Fig. 10
Fig. 10
Confusion Matrix on the validation set of the AFEW database
See this image and copyright information in PMC

References

    1. Fan Y, Lu X, Li D and Liu Y (2016) Video-based emotion recognition using cnn-rnn and c3d hybrid networks. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction, pp 445–450
    1. Hasani B and Mahoor MH (2017) Facial expression recognition using enhanced deep 3d convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp 30–40
    1. Xingjian S, Chen Z, Wang H, D.-Y. Yeung, W.-K. Wong, and W.-c. Woo, “Convolutional lstm network: A machine learning approach for precipitation nowcasting,” in Advances in neural information processing systems, 2015, pp. 802–810.
    1. Srivastava N, Mansimov E and Salakhudinov R (2015) Unsupervised learning of video representations using lstms. In: International conference on machine learning, pp 843–852
    1. Wang Z and Ying Z (2012) Facial expression recognition based on local phase quantization and sparse representation. In: 2012 8th International Conference on Natural Computation. IEEE, pp 222–225

LinkOut - more resources