Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 May 5:12:627561.
doi: 10.3389/fpsyg.2021.627561. eCollection 2021.

Automatic Facial Expression Recognition in Standardized and Non-standardized Emotional Expressions

Theresa Küntzler  1 , T Tim A Höfling  2 , Georg W Alpers  2
Affiliations

Automatic Facial Expression Recognition in Standardized and Non-standardized Emotional Expressions

Theresa Küntzler et al. Front Psychol. .

Abstract

Emotional facial expressions can inform researchers about an individual's emotional state. Recent technological advances open up new avenues to automatic Facial Expression Recognition (FER). Based on machine learning, such technology can tremendously increase the amount of processed data. FER is now easily accessible and has been validated for the classification of standardized prototypical facial expressions. However, applicability to more naturalistic facial expressions still remains uncertain. Hence, we test and compare performance of three different FER systems (Azure Face API, Microsoft; Face++, Megvii Technology; FaceReader, Noldus Information Technology) with human emotion recognition (A) for standardized posed facial expressions (from prototypical inventories) and (B) for non-standardized acted facial expressions (extracted from emotional movie scenes). For the standardized images, all three systems classify basic emotions accurately (FaceReader is most accurate) and they are mostly on par with human raters. For the non-standardized stimuli, performance drops remarkably for all three systems, but Azure still performs similarly to humans. In addition, all systems and humans alike tend to misclassify some of the non-standardized emotional facial expressions as neutral. In sum, emotion recognition by automated facial expression recognition can be an attractive alternative to human emotion recognition for standardized and non-standardized emotional facial expressions. However, we also found limitations in accuracy for specific facial expressions; clearly there is need for thorough empirical evaluation to guide future developments in computer vision of emotional facial expressions.

Keywords: automatic facial coding; facial expression recognition; human emotion recognition; naturalistic expressions; recognition of emotional facial expressions; software evaluation; specific emotions; standardized inventories.

PubMed Disclaimer

Conflict of interest statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Figures

Figure 1
Figure 1
Averaged human ratings separately for basic emotion categories for standardized (black bars) and non-standardized facial expressions (gray bars). (A) Depicts mean genuineness ratings ranging from 1 (very in-genuine) to 7 (very genuine). (B–H) Depict mean emotion ratings (percent) for (B) neutral, (C) joy, (D) anger, (E) disgust, (F) sadness, (G) fear, and (H) surprise expressions. Error bars are 95% confidence intervals.
Figure 2
Figure 2
Confusion matrices indicating classification performance on standardized (left panels) and non-standaridzed data (right panels): (A) standardized data by Azure, (B) non-standardized data by Azure, (C) standardized data by Face++, (D) non-standardized data by Face++, (E) standardized data by FaceReader and (F) non-standardized data by FaceReader. Numbers indicate percentages to the base of the true category. Reading example: From the standardized data Azure classifies 4.5% of the truly fearful expressions as neutral. The 45.5% of the fearful images are classified correctly.
Figure 3
Figure 3
Classification performance depicted as Receiver Operating Characteristic (ROC) curves and corresponding Area under the Curve (AUC) for overall emotion recognition performance for the three FER systems (Azure, Face++, and FaceReader) and human raters. Separately for (A) standardized facial expressions and (B) non-standardized facial expressions separately. The white diagonal line indicates classification performance by chance.
See this image and copyright information in PMC

References

    1. Abdullah S., Murnane E. L., Costa J. M. R., Choudhury T. (2015). Collective smile: measuring societal happiness from geolocated images, in CSCW '15: Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, Tallinn, Estonia. 361–374. 10.1145/2675133.2675186 - DOI
    1. Aggarwal A., Lohia P., Nagar S., Dey K., Saha D. (2019). Black box fairness testing of machine learning models, in ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Tallinn, Estonia. 625–635. 10.1145/3338906.3338937 - DOI
    1. Arriaga O., Valdenegro-Toro M., Plöger P. (2017). Real-time convolutional neural networks for emotion and gender classification. CoRR, abs/1710.07557.
    1. Barrett L. F., Adolphs R., Marsella S., Martinez A. M., Pollak S. D. (2019). Emotional expressions reconsidered: challenges to inferring emotion from human facial movements. Psychol. Sci. Publ. Interest 20, 1–68. 10.1177/1529100619832930 - DOI - PMC - PubMed
    1. Bartkiene E., Steibliene V., Adomaitiene V., Juodeikiene G., Cernauskas D., Lele V., et al. . (2019). Factors affecting consumer food preferences: food taste and depression-based evoked emotional expressions with the use of face reading technology. BioMed Res. Int. 2019:2097415. 10.1155/2019/2097415 - DOI - PMC - PubMed

LinkOut - more resources