Children with autism spectrum disorder produce more ambiguous and less socially meaningful facial expressions: an experimental study using random forest classifiers

Abstract

Background: Computer vision combined with human annotation could offer a novel method for exploring facial expression (FE) dynamics in children with autism spectrum disorder (ASD).

Methods: We recruited 157 children with typical development (TD) and 36 children with ASD in Paris and Nice to perform two experimental tasks to produce FEs with emotional valence. FEs were explored by judging ratings and by random forest (RF) classifiers. To do so, we located a set of 49 facial landmarks in the task videos, we generated a set of geometric and appearance features and we used RF classifiers to explore how children with ASD differed from TD children when producing FEs.

Results: Using multivariate models including other factors known to predict FEs (age, gender, intellectual quotient, emotion subtype, cultural background), ratings from expert raters showed that children with ASD had more difficulty producing FEs than TD children. In addition, when we explored how RF classifiers performed, we found that classification tasks, except for those for sadness, were highly accurate and that RF classifiers needed more facial landmarks to achieve the best classification for children with ASD. Confusion matrices showed that when RF classifiers were tested in children with ASD, anger was often confounded with happiness.

Limitations: The sample size of the group of children with ASD was lower than that of the group of TD children. By using several control calculations, we tried to compensate for this limitation.

Conclusion: Children with ASD have more difficulty producing socially meaningful FEs. The computer vision methods we used to explore FE dynamics also highlight that the production of FEs in children with ASD carries more ambiguity.

Keywords: Algorithm; Autism spectrum disorder; Emotion; Facial expressions.

Fig. 1

Framework of the computer vision method to explore FEs in TD children and children with ASD: experiment to induce FEs (a) and FE recognition pipeline (b)

Fig. 2

Boxplots: the solid line represents the median of the distribution; the outlines of the box represent the interquartile range, or 25th–75th percentiles; the whiskers represent the upper and lower quartiles, excluding outliers; and the diamonds represent the mean

Fig. 3

Facial landmarks contributing to the classification of happiness using RF classifiers (training and testing) in TD children (left) and children with ASD (a distance; b HOG)