Automated measurement of infant and mother Duchenne facial expressions in the Face-to-Face/Still-Face

Abstract

Although still-face effects are well-studied, little is known about the degree to which the Face-to-Face/Still-Face (FFSF) is associated with the production of intense affective displays. Duchenne smiling expresses more intense positive affect than non-Duchenne smiling, while Duchenne cry-faces express more intense negative affect than non-Duchenne cry-faces. Forty 4-month-old infants and their mothers completed the FFSF, and key affect-indexing facial Action Units (AUs) were coded by expert Facial Action Coding System coders for the first 30 s of each FFSF episode. Computer vision software, automated facial affect recognition (AFAR), identified AUs for the entire 2-min episodes. Expert coding and AFAR produced similar infant and mother Duchenne and non-Duchenne FFSF effects, highlighting the convergent validity of automated measurement. Substantive AFAR analyses indicated that both infant Duchenne and non-Duchenne smiling declined from the FF to the SF, but only Duchenne smiling increased from the SF to the RE. In similar fashion, the magnitude of mother Duchenne smiling changes over the FFSF were 2-4 times greater than non-Duchenne smiling changes. Duchenne expressions appear to be a sensitive index of intense infant and mother affective valence that are accessible to automated measurement and may be a target for future FFSF research.

Figure 1.

Expert coding and automated measurement of infant and mother Duchenne and non-Duchenne expressions in the first 30 seconds of each episode. Line plots contain estimated marginal means of the proportions of infant Duchenne and non-Duchenne smiling (first row), infant Duchenne and non-Duchenne cry-faces (second row), and mother Duchenne and non-Duchenne smiling (third row). The first column shows the Duchenne expressions, and the second column shows the non-Duchenne expressions. The x-axes indicate the FFSF episodes (FF, SF, and RE for infants; FF and RE for mothers). The y-axes indicate mean proportions. The blue line represents AFAR coding, and the red line represents expert FACS coding. Error bars indicate 95% confidence intervals.

Figure 2.

Full-episode automated measurement over the full FFSF. Estimated marginal means of the proportions of AFAR-identified infant Duchenne and non-Duchenne smiling (first plot), infant Duchenne and non-Duchenne cry-faces (second plot), and mother Duchenne and non-Duchenne smiling (third plot) in each full 2-minute episode of the FFSF. Time in infant smiling, infant cry-faces, and mother smiling as a proportion of time in each episode of the FFSF. The blue line represents Duchenne expressions, and the red line represents non-Duchenne expressions. The first plot demonstrates significant declines in both infant Duchenne and non-Duchenne smiling from the FF to the SF, and a rise in infant Duchenne smiling from the SF to the RE. The second plot illustrates a significant rise in infant Duchenne cry-faces from the FF to the SF and no change from the SF to the RE. There was no difference in infant non-Duchenne cry-faces between the episodes. The third plot shows significant declines in mother Duchenne and non-Duchenne smiling from the FF to the SF, and a rise from the SF to the RE. Error bars indicate 95% confidence intervals. Note that the y-axis scales are different for infant smiling, infant cry-faces, and mother smiling.