Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Apr 22;23(9):4204.
doi: 10.3390/s23094204.

Facial Expression Recognition Methods in the Wild Based on Fusion Feature of Attention Mechanism and LBP

Jun Liao  1   2   3 , Yuanchang Lin  1   3 , Tengyun Ma  1   3 , Songxiying He  3 , Xiaofang Liu  1   3 , Guotian He  1   3
Affiliations

Facial Expression Recognition Methods in the Wild Based on Fusion Feature of Attention Mechanism and LBP

Jun Liao et al. Sensors (Basel). .

Abstract

Facial expression methods play a vital role in human-computer interaction and other fields, but there are factors such as occlusion, illumination, and pose changes in wild facial recognition, as well as category imbalances between different datasets, that result in large variations in recognition rates and low accuracy rates for different categories of facial expression datasets. This study introduces RCL-Net, a method of recognizing wild facial expressions that is based on an attention mechanism and LBP feature fusion. The structure consists of two main branches, namely the ResNet-CBAM residual attention branch and the local binary feature (LBP) extraction branch (RCL-Net). First, by merging the residual network and hybrid attention mechanism, the residual attention network is presented to emphasize the local detail feature information of facial expressions; the significant characteristics of facial expressions are retrieved from both channel and spatial dimensions to build the residual attention classification model. Second, we present a locally improved residual network attention model. LBP features are introduced into the facial expression feature extraction stage in order to extract texture information on expression photographs in order to emphasize facial feature information and enhance the recognition accuracy of the model. Lastly, experimental validation is performed using the FER2013, FERPLUS, CK+, and RAF-DB datasets, and the experimental results demonstrate that the proposed method has superior generalization capability and robustness in the laboratory-controlled environment and field environment compared to the most recent experimental methods.

Keywords: LBP features; attention mechanism; deep learning; facial expression recognition.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Figure 1
Figure 1
Convolutional block attention module structure diagram.
Figure 2
Figure 2
Channel attention module diagram.
Figure 3
Figure 3
Spatial attention module diagram.
Figure 4
Figure 4
ResNet-CBAM Residual Attention Module. (a) Original residual module, (b) Residual attention module.
Figure 5
Figure 5
Basic network residual structure integration of CBAM and ResNet. CB—convolutional block; CAM—channel attention module; SAM—spatial attention module.
Figure 6
Figure 6
Face extraction and LBP feature extraction.
Figure 7
Figure 7
Locally enhanced residual network structure (RCL-Net).
Figure 8
Figure 8
Sample images from FER2013 dataset.
Figure 9
Figure 9
Sample images from FERPLUS dataset.
Figure 10
Figure 10
Sample images from CK+ dataset.
Figure 11
Figure 11
Sample images from RAF-DB dataset.
Figure 12
Figure 12
Confusion matrix for FER2013 test dataset.
Figure 13
Figure 13
Confusion matrix for FERPLUS test dataset.
Figure 14
Figure 14
Confusion matrix for CK+ test dataset.
Figure 15
Figure 15
Confusion matrix for RAF-DB test dataset.
Figure 16
Figure 16
Sample visualization results of different datasets, the left side is the original image graph and the right side is the visualization graph.
See this image and copyright information in PMC

References

    1. Li S., Deng W. Deep Facial Expression Recognition: A Survey. IEEE Trans. Affective Comput. 2022;13:1195–1215. doi: 10.1109/TAFFC.2020.2981446. - DOI
    1. Lucey P., Cohn J.F., Kanade T., Saragih J., Ambadar Z., Matthews I. The Extended Cohn-Kanade Dataset (CK+): A Complete Dataset for Action Unit and Emotion-Specified Expression; Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition—Workshops; San Francisco, CA, USA. 13–18 June 2010; pp. 94–101.
    1. Lyons M., Akamatsu S., Kamachi M., Gyoba J. Coding Facial Expressions with Gabor Wavelets; Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition; Nara, Japan. 14–16 April 1998; pp. 200–205.
    1. Valstar M.F., Pantic M. Induced Disgust, Happiness and Surprise: An Addition to the MMI Facial Expression Database; Proceedings of the 3rd International Workshop on Emotion; Paris, France. 29 October 2010.
    1. Zhao G., Huang X., Taini M., Li S.Z., Pietikäinen M. Facial Expression Recognition from Near-Infrared Videos. Image Vis. Comput. 2011;29:607–619. doi: 10.1016/j.imavis.2011.07.002. - DOI