Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023;35(20):14963-14972.
doi: 10.1007/s00521-023-08498-w. Epub 2023 Apr 1.

An automatic improved facial expression recognition for masked faces

Yasmeen ELsayed  1 , Ashraf ELSayed  1   2 , Mohamed A Abdou  3
Affiliations

An automatic improved facial expression recognition for masked faces

Yasmeen ELsayed et al. Neural Comput Appl. 2023.

Abstract

Automatic facial expression recognition (AFER), sometimes referred to as emotional recognition, is important for socializing. Automatic methods in the past two years faced challenges due to Covid-19 and the vital wearing of a mask. Machine learning techniques tremendously increase the amount of data processed and achieved good results in such AFER to detect emotions; however, those techniques are not designed for masked faces and thus achieved poor recognition. This paper introduces a hybrid convolutional neural network aided by a local binary pattern to extract features in an accurate way, especially for masked faces. The basic seven emotions classified into anger, happiness, sadness, surprise, contempt, disgust, and fear are to be recognized. The proposed method is applied on two datasets: the first represents CK and CK +, while the second represents M-LFW-FER. Obtained results show that emotion recognition with a face mask achieved an accuracy of 70.76% on three emotions. Results are compared to existing techniques and show significant improvement.

Keywords: Convolution neural network; Facial expression recognition; Feature extraction; Local binary pattern.

PubMed Disclaimer

Conflict of interest statement

Conflict of interestAuthors declare that there is neither funding nor conflict of interest.

Figures

Fig. 1
Fig. 1
Overall system block diagram
Fig. 2
Fig. 2
Facial landmarks points with Dlib library [32, 34]
Fig. 3
Fig. 3
An example of the basic LBP operator [35]
Fig. 4
Fig. 4
Hybrid LBP-CNN architecture on CK + dataset
Fig. 5
Fig. 5
Hybrid LBP-CNN architecture on M-LFW-FER dataset
Fig. 6
Fig. 6
Accuracy and loss curves of hybrid LBP-CNN model on CK + dataset
Fig. 7
Fig. 7
Accuracy and loss curves of proposed CNN (front + side view) model on M-LFW-FER dataset
Fig. 8
Fig. 8
Accuracy and loss curves of proposed CNN and LBP (front + side view) model on M-LFW-FER dataset
Fig. 9
Fig. 9
Accuracy and loss curves of proposed CNN (front view only) model on M-LFW-FER dataset
Fig. 10
Fig. 10
Accuracy and loss curves of proposed CNN and LBP (front view only) model on M-LFW-FER dataset
See this image and copyright information in PMC

References

    1. Yang B, Wu J, Hattori G (2021) Occlusion aware facial landmark detection based facial Ex- pression recognition with face mask
    1. Song J, Wei Y, Ke H. The effect of emotional information from eyes on empathy for pain: A subliminal ERP study. PLoS ONE. 2019 doi: 10.1371/journal.pone.0226211. - DOI - PMC - PubMed
    1. Manfred S (2020) Masked education? The benefits and burdens of wearing face masks in schools during the current Corona pandemic. Trends in Neuroscience and Education. 10.1016/j.tine.2020.100138 - PMC - PubMed
    1. Fuzail K (2019) Facial expression recognition using facial landmark detection and feature extraction via neural networks. arXiv preprint arXiv. 10.48550/arXiv.1812.04510
    1. Iacozza S, Costa A, Dun˜abeitia JA. What do your eyes reveal about your foreign language? Reading emotional sentences in a native and foreign language. IPloS one. 2017;12(10):e0186027. doi: 10.1371/journal.pone.0186027. - DOI - PMC - PubMed

LinkOut - more resources