A multimodal neural signature of face processing in autism within the fusiform gyrus

Abstract

Atypical face processing is commonly reported in autism. Its neural correlates have been explored extensively across single neuroimaging modalities within key regions of the face processing network, such as the fusiform gyrus (FFG). Nonetheless, it is poorly understood how variation in brain anatomy and function jointly impacts face processing and social functioning. Here we leveraged a large multimodal sample to study the cross-modal signature of face processing within the FFG across four imaging modalities (structural magnetic resonance imaging (MRI), resting-state functional magnetic resonance imaging, task-functional magnetic resonance imaging and electroencephalography) in 204 autistic and nonautistic individuals aged 7-30 years (case-control design). We combined two methodological innovations-normative modeling and linked independent component analysis-to integrate individual-level deviations across modalities and assessed how multimodal components differentiated groups and informed social functioning in autism. Groups differed significantly in a multimodal component driven by bilateral resting-state functional MRI, bilateral structure, right task-functional MRI and left electroencephalography loadings in face-selective and retinotopic FFG. Multimodal components outperformed unimodal ones in differentiating groups. In autistic individuals, multimodal components were associated with cognitive and clinical features linked to social, but not nonsocial, functioning. These findings underscore the importance of elucidating multimodal neural associations of social functioning in autism, offering potential for the identification of mechanistic and prognostic biomarkers.

Keywords: Autism spectrum disorders; Computational neuroscience.

Fig. 1. Overview of the methodological approach.

a, The features for each modality were extracted from the right and the left FFG. These were (1) GM volume based on VBM for structural MRI, (2) T-maps contrasting the faces condition to the shapes condition reflecting sensitivity to faces from the Hariri paradigm for task-fMRI, (3) seed-based connectivity analysis (SCA) between the FFA and all other intra-FFG voxels for rs-fMRI and (4) the principal component of source-reconstructed time series for EEG. b, Next, normative modeling was applied to each imaging modality using Bayesian linear regression. The depicted trajectories per modality are schematic and the actual modality-specific normative models are depicted in Extended Data Fig. 1. c, To model cross-subject individual-level variation, resulting Z-deviation maps per modality were statistically merged using LICA resulting in measures of modality contributions and subject loadings. d, Next, we tested for group differences in ICs and group separability using either multi- or unimodal ICs. e, Finally, we computed multivariate associations (that is, CCA) between subject loadings and clinical, cognitive measures related to either social–communicative or nonsocial features.

Fig. 2. Multimodal components and group differences in IC44.

a, Among all ICs, 11 were considered multimodal (Fig. 2a), with a single modality contribution of not more than 90%. b, The violin plots display the distribution of subject loadings (IC44) for autistic and NAI. IC44 showed a significant group difference with autistic individuals (N = 99) having higher contributions than NAI (N = 105). The box plot shows the median and the interquartile range (that is, the 25th and 75th percentiles) with whiskers extending to 1.5 times the interquartile range from the first and third quartiles. L, left; R, right. Source data

Fig. 3. Spatial and temporal characterization of IC44.

a, The spatial and temporal Z-maps thresholded at the 95th percentile of the different modalities associated with IC44. Positive values (yellow) depict positive loadings onto the IC where autistic individuals have higher deviations than NAI. The negative values (blue) depict negative loadings onto the IC where autistic individuals have lower deviations than NAI. Suprathreshold time points for EEG are depicted in red. b–e, The spatial overlap of suprathreshold voxels with a probabilistic functional atlas of the occipito-temporal cortex (that is, VIS atlas) for the positive loadings (that is, autism > NAI) (b and c) and negative loadings (that is, autism < NAI) (d and e), for the left (b and d) and right (c and e) hemispheres. f, The VIS atlas and its different category-selective subregions. g–j, The spatial overlap of suprathreshold voxels with the structural HOA (depicted in k) for the positive loadings (g and h) and the negative loadings (i and j), for the left (g and i) and the right (h and j) hemispheres. k, The HOA and its four FFG subregions (that is, anterior and posterior divisions of the temporal FC, temporal-occipital FC and occipital FG. CoS, collateral sulcus; FC, fusiform cortex; IOG, inferior occipital gyrus; IOS, inferior occipital sulcus; OTS, occipital temporal sulcus; pOTS, posterior OTS; VIS, probabilistic functional atlas of the occipito-temporal cortex; R, right; L, left. v1v, v2v, v3v are ventral topographic regions in early visual cortex. Source data

Fig. 4. CCA between the multimodal ICs and social–communicative features.

The multivariate association (that is, canonical correlation) was significant between the 11 multimodal ICs and the social–communicative features associated with autism. a, The loadings of each multimodal component contributing to the CCA mode. b, The canonical correlation scatter plot color coded by the highest contributing clinical feature (ADOS social affect). The x axis depicts the projected behavioral CCA variate (projection behavior) and the y axis the multimodal ICs CCA variates (projection brain). c, The modality contributions of the four ICs that contribute significantly to the CCA. d, The loadings of each social–communicative feature contributing to the CCA mode. The asterisks show the significant loadings. e, The spatial and temporal patterns of each imaging modality that are significantly correlated with the social–communicative features. These are based on the significant correlation values between the Z deviations of each imaging modality and the canonical imaging variate derived from the CCA. ADOS social, ADOS social affect subscale; ADI comm, ADI communication subscale; Vineland daily, Vineland daily living skills subscale; Vineland comm, Vineland communication subscale; Vineland soc, Vineland Socialization subscale; R, right; L, left.

Extended Data Fig. 1. Example forward models across the modalities.

Forward models are shown (estimated across all individuals) depicting the spatial and temporal representations of the voxel-wise / timepoint-wise normative model for each imaging modality per hemisphere. For imaging modalities (structure, task-fMRI, resting-state fMRI) the normative model of the peak activation voxel within the fusiform face area (FFA) is shown, whereas for EEG at timepoint 170 ms. The regression line depicts the predicted values between 7 and 30 years of age along with centiles of confidence (95th and 99th). The blue dots are the true values for nonautistic males in the KCL site, whereas the red dots are the true values for the autistic males in the KCL site. Given that the categorical covariates (sex, site) will render slightly different slopes, here, we only plotted the trajectories in males and in the largest acquisition site (KCL) for descriptive purposes. Abbreviations: FFA=fusiform face area; NAI=nonautistic individuals; L=left; R=right. Source data

Extended Data Fig. 2. Fifty independent components.

We identified 50 independent components using Linked ICA. These are ordered according to how multimodal they are (that is, multimodal index). Each color represents one of the eight feature maps (‘modalities’) fed into the model. Overall, across these, the right hemisphere (51.7%) and the left hemisphere (48.3%) showed equal contributions. Single modality contributions were as follows: EEG R (35.0%) > EEG L (33.2%) > rs-fMRI R (11.2%) > rs-fMRI L (9.6%) > task-fMRI R (3.5%) > task-fMRI L (3.4%) > structure L (2.1%) > structure R (2.1%). Abbreviations: IC=independent component; L=left; R=right. Source data

Extended Data Fig. 3. Support Vector Machine classification.

We ran a support vector machine (SVM) classification algorithm to test whether the multimodal independent components (ICs) outperformed the unimodal ICs in discriminating autistic from nonautistic individuals. a) Data are presented as the area under the receiver operating characteristic curve (AUC) along with the 95% confidence interval (CI) as a function of different thresholds between 85% to 99% that define whether an IC is multimodal or unimodal. b) Data are presented as the AUC along with the 95% CI when forcing uni- and multimodal features to have the same number of ICs in each fold. In the beginning (up to six ICs) there are no differences, which become apparent when increasing the number of ICs included as features. Abbreviations: IC=independent component; AUC= area under the receiver operating characteristic curve; CI=confidence interval.

Extended Data Fig. 4. Canonical correlation analysis between the multimodal ICs and nonsocial features.

The multivariate association (that is, canonical correlation) was not significant between the eleven multimodal independent components (ICs) and the nonsocial features associated with autism (r = 0.45, pFDR = 0.64). Panel a shows the loadings of each multimodal component contributing to the CCA mode, while panel b) shows the loadings of each nonsocial feature contributing to the CCA mode. Panel c shows the canonical correlation scatterplot color-coded by the highest contributing nonsocial feature (ADOS RRB). The x-axis depicts the projected behavioral CCA variate and the y-axis the multimodal ICs CCA variates. Abbreviations: IC=independent component; CCA=canonical correlation analysis.

Extended Data Fig. 5. Flowchart and exclusion criteria per imaging modality.

Each imaging modality had a different amount of data available. Different exclusion criteria within modality could apply to several subjects; reported numbers are thus intersecting. Abbreviations: FD=framewise displacement in mm.