Neural processing of asynchronous audiovisual speech perception

Abstract

The temporal synchrony of auditory and visual signals is known to affect the perception of an external event, yet it is unclear what neural mechanisms underlie the influence of temporal synchrony on perception. Using parametrically varied levels of stimulus asynchrony in combination with BOLD fMRI, we identified two anatomically distinct subregions of multisensory superior temporal cortex (mSTC) that showed qualitatively distinct BOLD activation patterns. A synchrony-defined subregion of mSTC (synchronous>asynchronous) responded only when auditory and visual stimuli were synchronous, whereas a bimodal subregion of mSTC (auditory>baseline and visual>baseline) showed significant activation to all presentations, but showed monotonically increasing activation with increasing levels of asynchrony. The presence of two distinct activation patterns suggests that the two subregions of mSTC may rely on different neural mechanisms to integrate audiovisual sensory signals. An additional whole-brain analysis revealed a network of regions responding more with synchronous than asynchronous speech, including right mSTC, and bilateral superior colliculus, fusiform gyrus, lateral occipital cortex, and extrastriate visual cortex. The spatial location of individual mSTC ROIs was much more variable in the left than right hemisphere, suggesting that individual differences may contribute to the right lateralization of mSTC in a group SPM. These findings suggest that bilateral mSTC is composed of distinct multisensory subregions that integrate audiovisual speech signals through qualitatively different mechanisms, and may be differentially sensitive to stimulus properties including, but not limited to, temporal synchrony.

Figure 1. Example Stimuli

Stimulus presentations included synchronous auditory and visual components (a) and asynchronous presentations with offsets ranging from 100 to 400 ms with both visual (b) and auditory (c) components presented first. In an additional paradigm run to account for stimulus presentation time, visual frame rates and auditory bits per second were increased to match the asynchronous stimulus presentation times (d). In order to ensure timing precision, 1000 timing offsets were measured for each stimulus asynchrony level, (with different colors representing each offset condition), and frequencies out of 1000 were calculated in 5 ms bins (e).

Figure 2. Behavioral results

Perceived synchrony rates were collected during a pre-scan behavioral session (a). Accuracy rates of a two-alternative, forced-choice semantic categorization task during fMRI scanning sessions (b). Error bars in both panels reflect between-subject standard deviations.

Figure 3. BOLD responses in S-mSTC

Averaged timecourses (a) and BOLD response amplitudes (b) across levels of asynchrony extracted from individual subject's synchrony-defined mSTC ROIs, as defined by a synchronous (blue) > asynchronous (red) contrast.

Figure 4. BOLD responses in B-mSTC

Averaged timecourses (a) and BOLD response amplitudes (b) across levels of asynchrony extracted from individual subject's bimodal mSTC ROIs, as defined by a conjunction of activations with unisensory-auditory and unisensory-visual stimulus presentations. Synchronous and asynchronous trials used to define the other sub-region of mSTC are labeled in blue and red, respectively, for comparison.

Figure 5. A synchrony-sensitive integrative network

Using a synchronous > asynchronous contrast, a network of regions sensitive to modulations of temporal synchrony was identified.

Figure 6. Bimodal mSTC defined by group and individual GLMs

Bimodal mSTC was defined as the conjunction of regions that exhibited significant activation with both unisensory-auditory and unisensory-visual stimulus. A group contrast a revealed unilateral activation in right B-mSTC (a). Individual analysis using the same contrast revealed bilateral B-mSTC activation (b). Right individually-defined ROIs showed greater anatomical homogeneity (with each color representing a different individual) compared to left ROIs, a result that may explain the lack of significant group activation in left mSTC.

Figure 7. Group-defined subregions of mSTC

Two distinct regions of mSTC were defined with group data. The first subregion, S-mSTC, was defined with a synchronous > asynchronous contrast (blue), while the second subregion, B-mSTC, was defined using a conjunction of two contrasts, audio-only > baseline and visual-only > baseline (orange). Each vertical white line on the coronal image represents a slice that can be seen in the sagittal orientation to the right.