Audiovisual temporal correspondence modulates human multisensory superior temporal sulcus plus primary sensory cortices

Abstract

The brain should integrate related but not unrelated information from different senses. Temporal patterning of inputs to different modalities may provide critical information about whether those inputs are related or not. We studied effects of temporal correspondence between auditory and visual streams on human brain activity with functional magnetic resonance imaging (fMRI). Streams of visual flashes with irregularly jittered, arrhythmic timing could appear on right or left, with or without a stream of auditory tones that coincided perfectly when present (highly unlikely by chance), were noncoincident with vision (different erratic, arrhythmic pattern with same temporal statistics), or an auditory stream appeared alone. fMRI revealed blood oxygenation level-dependent (BOLD) increases in multisensory superior temporal sulcus (mSTS), contralateral to a visual stream when coincident with an auditory stream, and BOLD decreases for noncoincidence relative to unisensory baselines. Contralateral primary visual cortex and auditory cortex were also affected by audiovisual temporal correspondence or noncorrespondence, as confirmed in individuals. Connectivity analyses indicated enhanced influence from mSTS on primary sensory areas, rather than vice versa, during audiovisual correspondence. Temporal correspondence between auditory and visual streams affects a network of both multisensory (mSTS) and sensory-specific areas in humans, including even primary visual and auditory cortex, with stronger responses for corresponding and thus related audiovisual inputs.

Figure 1.

Schematic illustration of stimulus sequences and setup. A, Illustrative examples of timing for sequences in vision (top row) and, in audition, for the audiovisual correspondence condition (i.e., perfectly synchronous sequence, with jittered arrhythmic timing, average rate of 4 Hz, and rectangular distribution of 2–10 Hz). B, This example illustrates the noncorresponding condition; the two streams still have comparable stimulus rate (and other temporal statistics) overall but are now highly unrelated (differently jittered arrhythmic sequences, with a protective minimal window of 100 ms separating visual and auditory onsets; see green dotted lines). C, Example visual stimuli are depicted. Participants maintained central fixation, whereas optic fibers at 18° eccentricity were illuminated to produce a red cross stimulus or a nonoverlapping green square stimulus, with successive alternation between these. The task was to monitor the central yellow fixation light-emitting diode for occasional brightening (indicated here by enlarged central yellow dot; duration of 1 ms and average occurrence of 0.1 Hz), with timing unrelated to the task-irrelevant auditory or visual streams.

Figure 2.

fMRI results: BOLD signal differences for corresponding minus noncorresponding audiovisual stimulation. Group effects in the following: A, contralateral multisensory superior temporal sulcus; B, contralateral early visual cortex; C, bilateral auditory cortex, with contralateral peak. Shown for the RVF and LVF groups (columns 1, 2 and 3, 4, respectively). The intersubject mean parameter estimates (SPM betas, proportional to percentage signal change) are plotted for contralateral mSTS, primary visual cortex, and primary auditory cortex (each plot in corresponding rows to the brain activations shown) from the subject-specific maxima used in the individual analyses, averaged across LVF and RVF groups, with mean Montreal Neurological Institute coordinates below each bar graph. Brackets linking pairs of bars in these graphs all indicate significant differences across those conditions (p < 0.05 or better).

Figure 3.

fMRI results: BOLD signal differences for corresponding minus noncorresponding audiovisual stimulation in an illustrative single subject. mSTS, visual cortex, and auditory cortex activations are shown, with the STS, the calcarine fissure, and Heschl's gyrus highlighted in blue on that individual's anatomical scan. Localization of the effects with respect to these anatomical landmarks was implemented in every individual.

Figure 4.

A–C, Combined results of corresponding minus noncorresponding audiovisual stimulation for LVF and RVF groups, with hemisphere flipping to pool results contralateral to the audiovisual coincidence, which thereby appear in the apparently left hemisphere here (see Materials and Methods). Overall activations for AVC > NC in the following: A, contralateral mSTS; B, contralateral early visual cortex; C, contralateral auditory cortex. D–F, Enhanced functional coupling of seeded mSTS (this seeded region shown in D as filled blue circle) with visual (E) and auditory (F) areas in the context of audiovisual temporal coincidence versus noncoincidence. Voxels showing significantly greater functional coupling with the STS seed for that context are highlighted in red.

Figure 5.

Overlap of observed activation, for audiovisual correspondence minus noncorrespondence with functional coupling results from one illustrative participant, with STS, calcarine fissure, and Heschl's gyrus draw in blue onto the individual's structural scan. STS activation was used as the seed for the PPI analysis, whereas regions in Heschl's gyrus and calcarine fissure show both increased activation for audiovisual correspondence minus noncorrespondence and also enhanced coupling with ipsilateral STS in the context of audiovisual temporal correspondence [this overlap was formally tested for each individual by a conjunction of PPI results and experimental AVC minus NC within areas that showed a sensory-specific effect (visual > auditory and auditory > visual, respectively)].

Figure 6.

Results of directed information transfer analysis for temporally corresponding minus noncorresponding audiovisual conditions (i.e., difference in the inferred directional information transfer, attributable to condition) between STS (indicated schematically with purple circle), calcarine fissure (V1, indicated schematically with red circle), and Heschl's gyrus (A1, schematic blue circle), with direction of information transfer indicated via colored arrows. Numbers by each arrow indicate the measured change in influence (larger = stronger for temporally corresponding than noncorresponding condition) in the direction of each colored arrow. Colored brackets link pairs of numbers showing significant differences between the impact of condition, indicating that one direction of influence changed more than the converse direction attributable to temporal correspondence (p < 0.05 or better). White brackets indicate no such significant differences [nonsignificant (n.s.)]. The absolute values for DIT measures matter less than the reliability of any differences because absolute values can depend on imaging parameters (Hinrichs et al., 2006).