Recognizing facial expressions of emotion amid noise: A dynamic advantage

Abstract

Humans communicate internal states through complex facial movements shaped by biological and evolutionary constraints. Although real-life social interactions are flooded with dynamic signals, current knowledge on facial expression recognition mainly arises from studies using static face images. This experimental bias might stem from previous studies consistently reporting that young adults minimally benefit from the richer dynamic over static information, whereas children, the elderly, and clinical populations very strongly do (Richoz, Jack, Garrod, Schyns, & Caldara, 2015, Richoz, Jack, Garrod, Schyns, & Caldara, 2018b). These observations point to a near-optimal facial expression decoding system in young adults, almost insensitive to the advantage of dynamic over static cues. Surprisingly, no study has yet tested the idea that such evidence might be rooted in a ceiling effect. To this aim, we asked 70 healthy young adults to perform static and dynamic facial expression recognition of the six basic expressions while parametrically and randomly varying the low-level normalized phase and contrast signal (0%-100%) of the faces. As predicted, when 100% face signals were presented, static and dynamic expressions were recognized with equal efficiency with the exception of those with the most informative dynamics (i.e., happiness and surprise). However, when less signal was available, dynamic expressions were all better recognized than their static counterpart (peaking at ∼20%). Our data show that facial movements increase our ability to efficiently identify emotional states of others under the suboptimal visual conditions that can occur in everyday life. Dynamic signals are more effective and sensitive than static ones for decoding all facial expressions of emotion for all human observers.

Figure 1.

(A) Static examples of one identity expressing the six basic emotions at different levels of phase signal. The rows represent the six basic expressions (anger, disgust, fear, happiness, sadness, and surprise), and the columns the different levels of signal (0%, 20%, 40%, 50%, 60%, 80%, and 100%). We adapted the stimuli with permission from Gold et al. (2013). Further illustrative video examples can be seen with the following links for the static condition—Movie 1 and the dynamic condition Movie 3. (B) A schematic illustration of the adaptive sampling approach.

Figure 2.

Raw data and curve fitting for dynamic and static conditions overall and for each expression separately. Dots represent raw data, while lines represent fitted curves. The vertical dotted lines mark the amount of signal needed to reach the ceiling point (i.e., 99% of the maximum performance). The horizontal dotted line indicates chance level. Triangles below the x axis mark the presence of a significant difference in terms of accuracy between dynamic and static conditions.

Figure 3.

Dynamic–static curve difference overall and across each expression separately as a function of signal percentage. Lines show the dynamic-static difference between fitted curves as a function of signal percentage. Shaded areas around the curve indicate the corrected CI. Red shading indicates that the CI does not include 0 (i.e., statistically significant), which is illustrated here as the horizontal dashed line. Gray shading indicates that the CI includes 0, and the difference is therefore nonsignificant. CI = confidence interval.

Figure 4.

Accuracy level at 100% of signal. Accuracy levels in the dynamic and static conditions are reported overall and for each expression independently. Error bars represent the 95% CI. The * indicates a significant difference between conditions based on the corrected CI. CI = confidence interval.

Figure 5.

Dynamic advantage across expressions and cross-expression comparisons. (A) Bar plots represent the magnitude of the maximum dynamic advantage for each expression. Error bars represent the corrected CI. (B) The matrix represents the cross-expression comparisons of the magnitude of the maximum dynamic advantage. The significance of the contrasts was determined using the corrected CI and are color coded according to different alpha thresholds. CI = confidence interval.

Figure 6.

Minimum amount of signal needed by observers to reach ceiling points. The minimum amount of signal needed to reach the ceiling points (i.e., 99% of the maximum recognition performance) in both dynamic and static conditions are reported overall and for each expression independently. Error bars represent the 95% CI. The start indicates significant differences between conditions based on the corrected CI. CI = confidence interval.

Figure 7.

Minimum amount of signal needed by observers to reach the ceiling point (i.e., 99% of their maximum performance). Each expression is displayed with the level of signal at its ceiling point (i.e., 99% of maximum performance). Top and bottom rows illustrate the static and dynamic conditions, respectively.

Figure 8.

Minimum amount of signal needed by observers to surpass chance level. The minimum amount of signal needed, surpass chance level in both dynamic and static conditions is reported overall and for each expression independently. Error bars represent the 95% CI. The * indicates significant differences between conditions based on the corrected CI. CI = confidence interval.

Figure 9.

Maximum slope across conditions for each expression and overall. The maximum slopes of the fitted curves in the dynamic and static conditions are reported overall and for each expression independently. Error bars represent the 95% CI. The * indicates significant differences between conditions based on the corrected CI. CI = confidence interval.