Multisensory Integration of Naturalistic Speech and Gestures in Autistic Adults

Abstract

Seeing the speaker often facilitates auditory speech comprehension through audio-visual integration. This audio-visual facilitation is stronger under challenging listening conditions, such as in real-life social environments. Autism has been associated with atypicalities in integrating audio-visual information, potentially underlying social difficulties in this population. The present study investigated multisensory integration (MSI) of audio-visual speech information among autistic and neurotypical adults. Participants performed a speech-in-noise task in a realistic multispeaker social scenario with audio-visual, auditory, or visual trials while their brain activity was recorded using EEG. The neurotypical group demonstrated a non-linear audio-visual effect in alpha oscillations, whereas the autistic group showed merely additive processing. Despite these differences in neural correlates, both groups achieved similar behavioral audio-visual facilitation outcomes. These findings suggest that although autistic and neurotypical brains might process multisensory cues differently, they achieve comparable benefits from audio-visual speech. These results contribute to the growing body of literature on MSI atypicalities in autism.

Keywords: EEG; audio‐visual speech; autism; iconic gestures; multisensory integration.

FIGURE 1

(A) Social situation: A videoconference with the teacher (top left) moderating a word game for three participants. (B) Schematic representation of the trial order. The gray waves in the speech bubble indicate pink noise to create an adverse listening condition. The time window of relevance was 0–1 s from the onset of the target word (in this example: “comemos”).

FIGURE 2

Mean accuracy (left panel) and reaction times (right panel) per group and condition. The error bars reflect standard error. The dashed line for accuracy marks the 25% chance of performance in the task.

FIGURE 3

Behavioral benefits in word recognition (left panel) and neural MSI effect (z‐scores) (right panel) are found across the groups.

FIGURE 4

Time‐frequency representations (TFRs) per condition and group. Time in seconds is plotted on the horizontal axis, and frequency in Hz on the vertical axis. The dashed rectangles mark the time of interest (TOI; 0–1 s) and frequency of interest (FOI; alpha; 8–13 Hz). The TFRs are plotted for the region of interest (ROI) in the alpha analysis (CP1, CP3, P1, P3, and P5). The heads show the topography of the selected TOI and FOI, and the dashed circles mark the electrodes in ROI. The main effect of group (AUT vs. NT) and multi‐ vs. unisensory comparisons (AV vs. A, AV vs. V) are represented.

FIGURE 5

Spatio‐temporal unfolding of MSI across bands. The colors of the topographical maps represent the difference of multisensory (AV) minus the sum of unisensory (A + V) mean power (μV²). The circles correspond to the locations of the electrodes, with the large white circles marking the corrected MSI effect at this site and the small white circles representing additional sites with MSI effects, which did not survive the FDR correction.