Comprehensive Social Trait Judgments From Faces in Autism Spectrum Disorder

Abstract

Processing social information from faces is difficult for individuals with autism spectrum disorder (ASD). However, it remains unclear whether individuals with ASD make high-level social trait judgments from faces in the same way as neurotypical individuals. Here, we comprehensively addressed this question using naturalistic face images and representatively sampled traits. Despite similar underlying dimensional structures across traits, online adult participants with self-reported ASD showed different judgments and reduced specificity within each trait compared with neurotypical individuals. Deep neural networks revealed that these group differences were driven by specific types of faces and differential utilization of features within a face. Our results were replicated in well-characterized in-lab participants and partially generalized to more controlled face images (a preregistered study). By investigating social trait judgments in a broader population, including individuals with neurodevelopmental variations, we found important theoretical implications for the fundamental dimensions, variations, and potential behavioral consequences of social cognition.

Keywords: autism spectrum disorder; face perception; facial feature; open data; open materials; preregistered; social trait judgment.

Fig. 1.

Social trait judgments from participants who self-reported a positive clinical diagnosis of autism spectrum disorder (SR-ASD) and neurotypicals. (a) Example stimuli ranked by average ratings from neurotypicals for each social trait. (b) Principal component analysis loadings of social traits on the first four principal components. Each column plots the strength of the loadings (x-axis, absolute value) across traits (y-axis). Color coding indicates the sign of the loading (orange for positive, purple for negative). Saturated colors highlight each trait’s most strongly correlated principal component. (c, d) Interrater consistency of each trait was estimated using (c) the intraclass correlation coefficient (ICC; McGraw & Wong, 1996) and (d) the Spearman’s correlation coefficient (ρ). Interrater consistency was first calculated between raters and averaged within each module and then averaged across modules. Participants with SR-ASD demonstrated lower interrater consistency for most of the traits: warm (two-tailed paired-samples t test across 10 rating modules), ICC: t(9) = 3.45, p = .0073, d = 1.28, 95% confidence interval (CI) = [0.009, 0.04]; Spearman: t(9) = 3.19, p = .011, d = 1.18, 95% CI = [0.02, 0.11]; competent, ICC: t(9) = 5.43, p = .00042, d = 2.49, 95% CI = [0.07, 0.02]; Spearman: t(9) = 5.78, p = .00027, d = 2.24, 95% CI = [0.04, 0.09]; practical, ICC: t(9) = 4.21, p = .0023, d = 1.81, 95% CI = [0.06, 0.19]; Spearman: t(9) = 3.26, p = .0099, d = 1.23, 95% CI = [0.02, 0.09]; feminine, ICC: t(9) = 5.19, p = .00057, d = 2.08, 95% CI = [0.004, 0.009]; Spearman: t(9) = 7.28, p = 4.66×10⁻⁵, d = 2.56, 95% CI = [0.12, 0.23]; strong, ICC: t(9) = 4.06, p = .0029, d = 1.11, 95% CI = [0.009, 0.03]; Spearman: t(9) = 4.09, p = .0027, d = 1.28, 95% CI = [0.03, 0.10]; and youthful, ICC: t(9) = 4.49, p = .0015, d = 2.09, 95% CI = [0.006, 0.02]; Spearman: t(9) = 5.76, p = .00027, d = 2.64, 95% CI = [0.09, 0.20]. (e) For aggregate ratings, participants with SR-ASD gave statistically different ratings for five traits (two-way repeated measures analysis of variance)—main effect of participant group: F(1, 5613) = 12.15, p = 5.20×10⁻⁴, η² = .005; main effect of trait: F(7, 5163) = 214.35, p = 6.46×10⁻²³⁵, η² = .24; interaction: F(7, 5613) = 4.27, p = 1.03×10⁻⁴, η² = .002: warm (two-tailed two-sample t test across participants), t(824) = 3.31, p = .00097, d = 0.23, 95% CI = [0.06, 0.22]; practical, t(802) = 3.13, p = .0018, d = 0.22, 95% CI = [0.06, 0.26]; feminine, t(736) = 2.65, p = .0082, d = 0.20, 95% CI = [0.03, 0.17]; strong, t(822) = 3.10, p = .0020, d = 0.22, 95% CI = [0.06, 0.25]; and youthful, t(829) = 4.47, p = 9.03×10⁻⁶, d = 0.31, 95% CI = [0.11, 0.28]. Error bars denote standard errors of the mean (± SEM) across rating modules. Asterisks indicate a significant difference between participants with SR-ASD and neurotypicals using two-tailed two-sample t test. *p < .05, **p < .01, ***p < .001, ****p < .0001. (f) Ratings for each face identity rank-ordered by mean ratings from neurotypicals (red for SR-ASD, blue for neurotypical). Error bars and error shades denote standard errors of the mean (± SEM) across rating modules. (g) Average ratings for the 10 identities with the highest ratings from neurotypicals. (h) Average ratings for the 10 identities with the lowest ratings from neurotypicals. (i) Difference in ratings between the top 10 and bottom 10 identities.

Fig. 2.

Features across faces that contribute to different trait ratings in self-reported autism spectrum disorder (SR-ASD) compared with neurotypicals. (a–d) Estimation of the rating density and identification of the discriminative regions in the feature space. By comparing observed (left) and permuted (middle) difference in ratings between groups, we could identify a region in the feature space (right) where the difference in ratings was significant (discriminative regions). These regions contain faces that are most discriminative for ratings between individuals with SR-ASD and neurotypicals (delineated by the outlines; also shown in panel e). Color coding shows density in arbitrary units (a.u.). Each color in the scatterplot represents a different identity. (a) Trait competent. (b) Trait practical. (c) Trait feminine. (d) Trait youthful. (e) Discriminative regions in the face feature space constructed by t-distributed stochastic neighbor embedding for the deep neural network (DNN) fully connected layer FC6. All stimuli are shown in this space. The feature dimensions are in arbitrary units (a.u.). Outlines delineate the discriminative regions for each trait. (f, g) Representation similarity between social trait judgment ratings and DNN features for each DNN layer. Solid circles represent a significant above-chance correlation (permutation test: p < .05, Bonferroni correction across layers). Shaded area denotes standard deviation (± SD) across rating modules. Dashed line denotes standard deviation (± SD) across permutation runs. Asterisks indicate a significant difference between participants with SR-ASD and neurotypicals using permutation test (red for SR-ASD, blue for neurotypical). ***p < .001. (f) Trait practical. (g) Trait youthful.

Fig. 3.

Features within faces that contribute to different trait ratings in self-reported autism spectrum disorder (SR-ASD) compared with neurotypicals. Relevance of each pixel to classification was revealed using layer-wise relevance propagation (LRP). Color coding shows LRP values in arbitrary units (a.u.). Yellow pixels positively contributed to the classification, whereas blue pixels negatively contributed to the classification. (a, b) Two example faces and their corresponding LRP maps. (a) Trait warm. (b) Trait strong. (c) Average LRP maps for each trait and each group. Images from neurotypicals (upper). Images from participants with SR-ASD (lower). (d) Difference in LRP maps for each trait. Red contours show the regions with a significant difference between participants with SR-ASD and neurotypicals using two-tailed paired-samples t test (p < 10⁻¹⁸; cluster size > 5% of all pixels).

Fig. 4.

Validation with in-lab participants. (a) Principal component analysis loadings of social traits on the first four principal components. (b, c) Interrater consistency. Participants with ASD demonstrated lower interrater consistency for most of the traits: warm (two-tailed paired-samples t test across 10 rating modules), intraclass correlation coefficient (ICC): t(9) = 8.77, p = 1.05×10⁻⁵, d = 2.67, 95% confidence interval (CI) = [0.05, 0.08]; Spearman: t(9) = 13.55, p = 2.72×10⁻⁷, d = 4.50, 95% CI = [0.19, 0.27]; critical, Spearman: t(9) = 4.57, p = .001, d = 1.68, 95% CI = [0.05, 0.14]; competent, ICC: t(9) = 7.63, p = 3.22×10⁻⁵, d = 2.87, 95% CI = [0.15, 0.27]; Spearman: t(9) = 10.50, p = 2.39×10⁻⁶, d = 4.17, 95% CI = [0.10, 0.16]; practical, ICC: t(9) = 7.12, p = 5.54 ×10⁻⁵, d = 2.68, 95% CI = [0.10, 0.20]; Spearman: t(9) = 8.08, p = 2.05×10⁻⁵, d = 3.64, 95% CI = [0.09, 0.16]; feminine, ICC: t(9) = 2.80, p = .02, d = 1.32, 95% CI = [0.002, 0.02]; Spearman: t(9) = 5.21, p = 5.78×10⁻⁴, d = 2.64, 95% CI = [0.12, 0.29]; strong, ICC: t(9) = 2.90, p = .02, d = 1.11, 95% CI = [0.009, 0.07]; Spearman: t(9) = 5.32, p = 4.80×10⁻⁴, d = 2.45, 95% CI = [0.11, 0.26]; youthful, ICC: t(9) = 6.76, p = 8.26×10⁻⁵, d = 2.21, 95% CI = [0.02, 0.04]; Spearman: t(9) = 12.34, p = 6.05×10⁻⁷, d = 5.30, 95% CI = [0.19, 0.27]; and charismatic, ICC: t(9) = 6.95, p = 6.67×10⁻⁵, d = 2.14, 95% CI = [0.07, 0.14]; Spearman: t(9) = 12.47, p = 5.53×10⁻⁷, d = 2.86, 95% CI = [0.14, 0.20]. (d) For aggregate ratings, neurotypicals had a significantly higher rating for warm, t(379) = 3.12, p = .002, d = 0.32, 95% CI = [0.08, 0.35]; competent, t(350) = 4.28, p = 2.41×10⁻⁵, d = 0.47, 95% CI = [0.18, 0.50]; practical, t(365) = 4.41, p = 4.79×10⁻⁶, d = 0.49, 95% CI = [0.22, 0.53]; strong, t(316) = 3.30, p = .001, d = 0.34, 95% CI = [0.04, 0.18]; and charismatic, t(373) = 5.99, p = 4.95×10⁻⁹, d = 0.62, 95% CI = [0.28, 0.55]. (e) Ratings for each face identity rank-ordered by mean ratings from neurotypicals (red for ASD, blue for neurotypical). Error bars and error shades denote standard errors of the mean (± SEM) across rating modules. (f) Average ratings for the 10 identities with the highest ratings from neurotypicals. (g) Average ratings for the 10 identities with the lowest ratings from neurotypicals. (h, i) Difference in ratings between the top 10 and bottom 10 identities. Legend conventions as in Fig. 1.

Fig. 5.

Validation with an independent sample of participants using unfamiliar face stimuli. (a) Example stimuli ranked by average ratings from neurotypicals for each social trait. (b) Principal component analysis loadings of social traits on the first four principal components. (c, d) Interrater consistency. (c) Intraclass correlation coefficient (ICC). Participants with self-reported autism spectrum disorder (SR-ASD) demonstrated a lower ICC for warm (one-tailed two-sample t test across participant pairs), t(60769) = 2.40, p = .0082, d = 0.020, lower bound of 95% confidence interval (CI) = 0.0014; feminine, t(59782) = 25.6, p < 10⁻¹⁰, d = 0.21, lower bound of 95% CI = 0.03; strong, t(61508) = 27.0, p < 10⁻¹⁰, d = 0.22, lower bound of 95% CI = 0.07; and youthful, t(60516) = 15.5, p < 10⁻¹⁰, d = 0.13, lower bound of 95% CI = 0.05. (d) Spearman’s correlation coefficient (ρ). Participants with SR-ASD demonstrated a lower correlation coefficient for feminine, t(59782) = 21.4, p < 10⁻¹⁰, d = 0.17, lower bound of 95% CI = 0.03; strong, t(61504) = 28.7, p < 10⁻¹⁰, d = 0.23, lower bound of 95% CI = 0.05; and youthful, t(60516) = 23.2, p < 10⁻¹⁰, d = 0.19, lower bound of 95% CI = 0.04. (e) For aggregate ratings, participants with SR-ASD had a significantly higher rating for critical (one-tailed two-sample t test), t(489) = 1.83, p = .034, d = 0.09, higher bound of 95% CI = 0.009. (f) Ratings for each face identity rank-ordered by mean ratings from neurotypicals. (g) Average ratings for the 10 identities with the highest ratings from neurotypicals. (h) Average ratings for the 10 identities with the lowest ratings from neurotypicals. (i) Difference in ratings between the top 10 and bottom 10 identities. Legend conventions as in Fig. 1.