Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Review
. 2016 Aug;38(8):1548-68.
doi: 10.1109/TPAMI.2016.2515606. Epub 2016 Jan 7.

Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications

Ciprian Adrian Corneanu, Marc Oliu Simon, Jeffrey F Cohn, Sergio Escalera Guerrero
Review

Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications

Ciprian Adrian Corneanu et al. IEEE Trans Pattern Anal Mach Intell. 2016 Aug.

Abstract

Facial expressions are an important way through which humans interact socially. Building a system capable of automatically recognizing facial expressions from images and video has been an intense field of study in recent years. Interpreting such expressions remains challenging and much research is needed about the way they relate to human affect. This paper presents a general overview of automatic RGB, 3D, thermal and multimodal facial expression analysis. We define a new taxonomy for the field, encompassing all steps from face detection to facial expression recognition, and describe and classify the state of the art methods accordingly. We also present the important datasets and the bench-marking of most influential methods. We conclude with a general discussion about trends, important questions and future lines of research.

PubMed Disclaimer

Figures

Fig. 1:
Fig. 1:
In the 19th century, Duchenne de Boulogne conducted experiments on how FEs are produced. From [4].
Fig. 2:
Fig. 2:
Primary emotions expressed on the face. From left to right: disgust, fear, joy, surprise, sadness, anger. From [14].
Fig. 3:
Fig. 3:
Taxonomy for AFER in Computer Vision. Red corresponds to RGB, green to 3D, and purple to thermal.
Fig. 4:
Fig. 4:
Examples of lower and upper face AUs in FACS. Reprinted from [14].
Fig. 5:
Fig. 5:
Sample images from the LFPW dataset aligned with the Supervised Descent Method (SDM). Obtained from [81].
Fig. 6:
Fig. 6:
General execution pipeline for the different modality fusion approaches. The tensor product symbols represent the modality fusion strategy. Approach-specific components of the pipeline are represented with different line types: dotted corresponds to early fusion, dashed to late fusion, dashed-dotted to direct data fusion and gray to sequential fusion.
Fig. 7:
Fig. 7:
FE datasets. (a) The CK [190] dataset (top) contains posed exaggerated expressions. The CK+ [191] (bottom) extends CK by introducing spontaneous expressions. (b) MMI [192], the first dataset to contain profile views. (c) MultiPIE [193] has multiview samples under varying illumination conditions. (d) SFEW [194], an in the wild dataset. (e) Primary FEs in Bosphorus [195], a 3D dataset. (f) KTFE [196] dataset, thermal images of primary spontaneous FEs.
Fig. 8:
Fig. 8:
Historical evolution of AFER.
See this image and copyright information in PMC

References

    1. Roger Highfield RW and Jenkins R, “How your looks betray your personality,” New Scientist, 2009.
    1. Chastel A, Leonardo on Art and the Artist. Courier Corporation, 2002.
    1. Greenblatt S et al., “Toward a universal language of motion: reflections on a seventeenth century muscle man,” 1994.
    1. de Boulogne G-BD and Cuthbertson RA, The Mechanism of Human Facial Expression. Cambridge University Press, 1990.
    1. Darwin C, The expression of emotion in man and animals. Oxford University Press, 1872.