Cross-modal processing in early visual and auditory cortices depends on expected statistical relationship of multisensory information

Abstract

Previous studies have shown that processing information in one sensory modality can either be enhanced or attenuated by concurrent stimulation of another modality. Here, we reconcile these apparently contradictory results by showing that the sign of cross-modal interactions depends on whether the content of two modalities is associated or not. When concurrently presented auditory and visual stimuli are paired by chance, cue-induced preparatory neural activity is strongly enhanced in the task-relevant sensory system and suppressed in the irrelevant system. Conversely, when information in the two modalities is reliably associated, activity is enhanced in both systems regardless of which modality is task relevant. Our findings illustrate an ecologically optimal flexibility of the neural mechanisms that govern multisensory processing: facilitation occurs when integration is expected, and suppression occurs when distraction is expected. Because thalamic structures were more active when the senses needed to operate separately, we propose them to serve gatekeeper functions in early cross-modal interactions.

Figure 1.

Experimental paradigm. A blue or red peripheral frame served as cue and instructed subjects whether the auditory (red frame) or the visual (blue frame) target stimulus would be relevant.

Figure 2.

A, Regions identified as early auditory and visual cortex in a single inflated hemisphere of one representative subject. B, Event-related BOLD response in response to cue and target in the same subject (8 s trials only). C, Bar plots of group-averaged results.

Figure 3.

Activation clusters induced by the target and cue (group analysis). Note that the two stimuli activated different subareas in primary visual cortex.

Figure 4.

Mean activity averaged across early sensory areas (visual and auditory) as a function of whether the cued modality matched the sensory cortex or not.

Figure 5.

A, Regions identified as thalamus in a single subject. B, Bar plots of group averaged results.