The spread of attention across modalities and space in a multisensory object

Abstract

Attending to a stimulus is known to enhance the neural responses to that stimulus. Recent experiments on visual attention have shown that this modulation can have object-based characteristics, such that, when certain parts of a visual object are attended, other parts automatically also receive enhanced processing. Here, we investigated whether visual attention can modulate neural responses to other components of a multisensory object defined by synchronous, but spatially disparate, auditory and visual stimuli. The audiovisual integration of such multisensory stimuli typically leads to mislocalization of the sound toward the visual stimulus (ventriloquism illusion). Using event-related potentials and functional MRI, we found that the brain's response to task-irrelevant sounds occurring synchronously with a visual stimulus from a different location was larger when that accompanying visual stimulus was attended versus unattended. The event-related potential effect consisted of sustained, frontally distributed, brain activity that emerged relatively late in processing, an effect resembling attention-related enhancements seen at earlier latencies during intramodal auditory attention. Moreover, the functional MRI data confirmed that the effect included specific enhancement of activity in auditory cortex. These findings indicate that attention to one sensory modality can spread to encompass simultaneous signals from another modality, even when they are task-irrelevant and from a different location. This cross-modal attentional spread appears to reflect an object-based, late selection process wherein spatially discrepant auditory stimulation is grouped with synchronous attended visual input into a multisensory object, resulting in the auditory information being pulled into the attentional spotlight and bestowed with enhanced processing.

Fig. 1.

Stimuli and task. In different runs, subjects covertly directed attention to the stream of visual stimuli on one side of the monitor while ignoring all visual stimuli on the opposite side and all auditory stimuli. Visual stimuli were unilateral streams of flashing checkerboards randomly presented on each side at stimulus onset asynchronies (SOAs) of 350-650 ms. The subjects' task was to detect target stimuli (checkerboards with two dots) occurring infrequently (14%) at the attended location. Half of the visual stimuli were accompanied by a simultaneous task-irrelevant tone presented centrally.

Fig. 2.

Visual attention effects on unisensory visual standards. (a) ERPs effects: Topographic plots of the attention effects in the left and right stream. For each side, the P1 effect (90-120 ms) is most prominent at contralateral occipital electrodes. It is followed by the N1 effect, first at contralateral fronto-central sites (150-200 ms) and later at parietal sites (225-250 ms). (b) fMRI effects: Event-related activation maps (Left) show the responses in contralateral visual cortex for left and right visual standards, collapsed across attended and unattended conditions. These activations were then used as ROIs for analyzing the effects of attention (i.e., response amplitude for attended vs. unattended visual standards), which revealed enhanced activity contralaterally (Right).

Fig. 3.

Multisensory object-based attention effect on the task-irrelevant midline tones, collapsed across the attend-left and attend-right conditions. (a) Sequence of ERP subtractions leading to the isolation of the multisensory attention effects for the tones, shown at frontal site Fz. (Top) Attended condition: A mixture of auditory and visual components can be seen in the ERP to the combined audiovisual events (thick trace). The subtraction of the visual-alone ERP (dotted trace) from the audiovisual ERP yields the “extracted” ERPs to the tones in the context of an attended visual stimulus (thin solid trace). (Center) Unattended condition: The analogous subtraction is performed on the unattended unisensory visual standards (dotted trace) and the unattended visual standards paired with a central tone (thick trace). The thin solid trace shows the corresponding unattended-condition difference wave of the multisensory minus unisensory-visual ERPs. (Bottom) The extracted difference waves overlaid, revealing the multisensory attention effect on the synchronous tones. An attention-related difference, a frontally distributed processing negativity, emerges ≈220 ms and lasts for hundreds of milliseconds. (b) Multisensory object-based attention effect, for a number of electrode sites, laid out as on the subject's head. Although there is no difference between the attention conditions on the early auditory components (P20-50, N1), a sustained attentional difference starts to emerge at ≈220 ms over fronto-central and frontal sites and lasts for >400 ms. (c) Topographic voltage maps for the multisensory attention effect on the spatially discrepant tones. Shown are the difference maps for tones in the context of an attended vs. unattended lateral visual stimulus (see corresponding ERP waves in b). This attentional difference is maximal over fronto-central and frontal sites, similar to that of the attention-related N1 effect and processing negativity elicited by attended auditory stimuli in auditory attention experiments.

Fig. 4.

fMRI response showing the multisensory attentional context effects on the task-irrelevant midline tones in auditory cortex, collapsed across the attend-left and attend-right conditions. (Left) Extracted event-related tone responses, shown separately for when synchronous with an attended vs. an unattended lateral visual stimulus. (Middle) Extracted event-related tone response, collapsed across attention conditions, to obtain auditory cortex ROIs. (Right) Extracted event-related response amplitudes in these auditory cortex ROIs for each of the multisensory attention context conditions.