Multisensory interplay reveals crossmodal influences on 'sensory-specific' brain regions, neural responses, and judgments

Abstract

Although much traditional sensory research has studied each sensory modality in isolation, there has been a recent explosion of interest in causal interplay between different senses. Various techniques have now identified numerous multisensory convergence zones in the brain. Some convergence may arise surprisingly close to low-level sensory-specific cortex, and some direct connections may exist even between primary sensory cortices. A variety of multisensory phenomena have now been reported in which sensory-specific brain responses and perceptual judgments concerning one sense can be affected by relations with other senses. We survey recent progress in this multisensory field, foregrounding human studies against the background of invasive animal work and highlighting possible underlying mechanisms. These include rapid feedforward integration, possible thalamic influences, and/or feedback from multisensory regions to sensory-specific brain areas. Multisensory interplay is more prevalent than classic modular approaches assumed, and new methods are now available to determine the underlying circuits.

Figure 1

Response Properties of Multisensory Neurons (A) Response properties of a putatively illustrative multisensory neuron, in deep superior colliculus, which in this case shows the often-discussed nonlinearly superadditive pattern of firing. That is, the response for combined visual and auditory stimulation, with a particular spatiotemporal relation, greatly exceeds the sum of the responses to each modality alone (adapted from Stein et al., 2004, by permission of Oxford University Press). (B) Distribution of z scores for a population of sampled neurons within deep layers of the cat superior colliculus, where z scores relate to firing rates for combined audiovisual stimulation, as compared with summed unisensory auditory and unisensory visual responses (©2007 by Oxford University Press, reprinted with permission). Note that while the z scores are distributed, with some neurons showing nonlinear multisensory responses (shaded columns), as often emphasized in the literature and as exemplified in (A), the distribution does in fact appear normal around zero, indicating that the average (and majority) population response of SC neurons may be additive/linear, even though some individual neurons depart from this (adapted from Stein et al., 2004). See later text for possible implications for fMRI research, where nonlinear criteria have often been proposed for assessing multisensory population responses, but may be overly restrictive.

Figure 2

Anatomy of Cortical Multisensory Areas (A) Schematic overview of the anatomy of some cortical multisensory areas derived from anatomical, electrophysiological, and functional imaging data in nonhuman primates. (B) Illustration of candidate human multisensory cortical regions (found within prefrontal, parietal, and premotor cortex, plus superior temporal sulcus), derived from the overlap of BOLD responses to passive unisensory stimulation with brief auditory, visual, or tactile events in 12 healthy adult subjects, shown as a surface rendering for the left hemisphere. This reflects a new analysis of data from one experiment in our own fMRI work, but similar areas are implicated in many other human fMRI studies (e.g., see Macaluso and Driver, 2005). Cortical regions where visual and auditory responses overlap are shown in blue; those where visual and tactile responses overlap are shown in green; and regions showing a response to passive stimulation in any of these three modalities are shown in red. Similar regions also activated in right hemisphere. Because these depicted activations reflect BOLD responses induced merely by “simple” stimulation with brief events in one or another modality, we also depict (more schematically) additional cortical areas reported for combined multisensory stimuli, including face-voice combinations (Kriegstein et al., 2005), multisensory speech perception (Buchel et al., 1998), plus an area involved in visual-tactile shape interactions in lateral-occipital complex (LOC; Amedi et al., 2002).

Figure 3

Electrophysiological Effects of Visual and Auditory Stimulation in Macaque Auditory Cortex Illustration of the laminar current-source densities (CSDs), found in a subregion of auditory association cortex posterior-lateral to A1 in monkeys, due to visual and auditory stimulation when recorded with multicontact electrodes (intercontact distance 150 μm) (reprinted from Schroeder and Foxe, 2002, by permission of Elsevier). CSDs reflect local postsynaptic potential (PSP) patterns. In the CSD profile, downward deflections (dark shaded) signify net extracellular current sinks (representing inward transmembrane currents) while upward deflections (gray shaded) indicate net extracellular current sources (representing outward currents). Sinks and sources are associated with local de- or hyperpolarization in local neuronal ensembles, respectively. Blue boxes emphasize CSD configurations due to auditory stimuli that reflect the initial excitatory response at layer 4. Red boxes reflect CSD configurations due to visual stimuli above and below layer 4 (see also illustrative diagram of feedforward/feedback connections in leftmost column overlaid on the six layers of auditory association cortex). These results strongly suggest that both auditory and visual stimuli are processed in this particular “auditory” area. However, the underlying neural mechanisms are different and indicative of feedforward versus feedback processing, respectively.

Figure 4

Multisensory Interactions in Humans Illustration of Noesselt et al. (2007) human fMRI study on audiovisual correspondence in temporal pattern (©2007 by the Society for Neuroscience, reprinted with permission). Schematics illustrating the stimulus set-up and design are shown at left, with illustrative group fMRI results on the right. (A) The top-left schematic illustrates a series of peripheral visual transients (change from green square to red cross, implemented inside the scanner with optic fibers) in the upper-right visual quadrant, while the participant fixates the lower central dot throughout, monitoring that for an occasional change in its brightness. During the stream of peripheral visual transients, a stream of auditory sound bursts (not shown in top schematic) could be emitted from a loudspeaker above the fixation point inside the scanner. (B and C) As shown in the two timeline schematics, visual and auditory streams each had erratic timing, and when both were present they either corresponded perfectly with each other (coincident temporal patterns, as in [B]) or had no temporal correspondence (as in [C]) despite comparable temporal statistics overall. (D) Relative to unimodal conditions (i.e., just visual or just auditory streams), audiovisual temporal correspondence (which is highly unlikely to arise by chance alone for these erratic temporal patterns) increased BOLD signal in superior temporal sulcus (STS, top brain image), contralateral to the corresponding visual stream (blue-green activation shown arises when that stream was in the right visual field, red-yellow activation when in the opposite visual field), whereas noncorrespondence decreased BOLD signal relative to the same unimodal baselines. Remarkably, an analogous pattern of results was also found for visual and auditory cortex (middle and bottom brain images), including primary areas (V1 and A1), even when considered at the level of each individual participant. Moreover, analyses of functional coupling and of directed information transfer between areas, for the BOLD data, indicated an influence from STS upon V1 and A1 that was significantly enhanced for the temporally corresponding condition, consistent with a possible feedback influence from STS.

Figure 5

Possible Neural Pathways Mediating Multisensory Interplay, Shown Schematically to Make the Abstract Possibilities Discussed in the Main Text More Concrete (A) Direct feedforward influences between visual and auditory processing, which might either arise subcortically at thalamic levels, as sketched in (I), if multisensory (MS) thalamus influences visual cortex (VC); and/or via sparse cortical-cortical connections directly between auditory cortex (AC, blue), visual cortex (VC, red), and somatosensory or tactile cortex (TC, yellow), as in (II). (B) Some multisensory regions may exist near classic unisensory regions, as for some audio-visual areas (violet) and some audio-tactile (green) areas near conventional auditory cortex (blue). (C) Feedback connections may exist from higher-level multisensory regions, back to lower-level areas that are (predominantly) sensory specific apart from these feedback influences. For instance, visual and tactile modalities may interact via particular regions of posterior parietal cortex (PP, orange) that receive afferent input from both modalities and send feedback projections to each; and analogously, auditory and visual modalities may interact in posterior STS (violet) and send feedback projections to sensory-specific auditory and visual cortex. As discussed in the main text, while such potential architectures are often considered as rival views, in fact all of them may coexist. Future work needs to identify which particular pathways/architectures are causally involved in particular multisensory effects.