Multisensory Integration and the Society for Neuroscience: Then and Now

Abstract

The operation of our multiple and distinct sensory systems has long captured the interest of researchers from multiple disciplines. When the Society was founded 50 years ago to bring neuroscience research under a common banner, sensory research was largely divided along modality-specific lines. At the time, there were only a few physiological and anatomical observations of the multisensory interactions that powerfully influence our everyday perception. Since then, the neuroscientific study of multisensory integration has increased exponentially in both volume and diversity. From initial studies identifying the overlapping receptive fields of multisensory neurons, to subsequent studies of the spatial and temporal principles that govern the integration of multiple sensory cues, our understanding of this phenomenon at the single-neuron level has expanded to include a variety of dimensions. We now can appreciate how multisensory integration can alter patterns of neural activity in time, and even coordinate activity among populations of neurons across different brain areas. There is now a growing battery of sophisticated empirical and computational techniques that are being used to study this process in a number of models. These advancements have not only enhanced our understanding of this remarkable process in the normal adult brain, but also its underlying circuitry, requirements for development, susceptibility to malfunction, and how its principles may be used to mitigate malfunction.

Figure 1.

The rapid growth of interest in multisensory integration. Left, Number of research articles indexed by the key word “multisensory” (on PubMed) published each year since the inception of the annual meeting in 1971. Right, Number of multisensory-related abstracts at the annual meeting of the Society for Neuroscience (years 2008–2015), including key words multisensory, polysensory, intersensory, cross-modal, heteromodal, multimodal, polymodal, supramodal, and amodal.

Figure 2.

The cat model used to study the multisensory principles of SC neurons. Top left, The overlapping visual and auditory receptive fields of a multisensory SC neuron on a polar map of visual-auditory space. Below it are impulse rasters showing the neuron's unisensory (V, Visual; A, auditory) and multisensory (VA) responses. Bar graph below them represents the magnitude of the multisensory enhancement evoked by their combination (ME, the proportionate amplification relative to the best unisensory response). Error bars indicate s.e.m., dashed line indicates the sum of the unisensory response magnitudes. Top right, This physiological enhancement facilitates the detection, localization, and orientation roles of the structure. Bottom, A perimetry device with LEDs and speakers used to probe multisensory behavioral enhancement.

Figure 3.

Multisensory integration enhances performance in a number of perceptual and behavioral domains.

Figure 4.

The physiological bases of multisensory integration can be measured in different ways, and can manifest differently in different brain regions. Left to right, Inputs from another modality (S) can shift the phase of ongoing background oscillations to resonate with an incoming signal (A). “Unisensory” neurons in different brain regions can synchronize their activity to amplify the impact of their signals on target structures. Individual multisensory neurons can integrate incoming signals as soon as they arrive to amplify responses. Event-related potential and local field potential methods are used to detect gross changes in responses at the ensemble level. Multisensory integration can improve discrimination by producing more reliable distributions of activity in a feature map, and can lead to more coherent activation patterns within large-scale networks, reflecting more efficient information processing.

Figure 5.

Computational modeling of multisensory integration. Left, Bayesian frameworks describe how ambiguous sensory signals can be combined with prior expectations that the information being offered refers to the same event to form optimal multisensory estimates. Depicted are variable estimates of a sensory feature from each modality that combine to form a joint distribution, which is then combined with a prior distribution peaked along the diagonal, representing an assumption that these signals have a common cause. The product is a distribution of multisensory estimates that represents an optimal combination of the unisensory inputs and is more reliable than either alone. Middle, Network models of multisensory integration have increased in sophistication from simple, abstract architectures involving three areas (e.g., two unisensory areas and one multisensory area) to models that include multiple biologically realistic inputs. In this diagram, ovals represent processing areas containing multiple units (circles). There are a total of four modeled input areas: two derived from cortical regions AES (AEV, visual; FAES, auditory) and two derived from non-AES sources (V, visual; A, auditory). These areas extend projections to integrating neurons in the SC. Right, Models of single units performing multisensory integration no longer seek to describe the responses of a “canonical” or “average” multisensory product calculated over a wide window of time but can successfully predict the responses of individual neurons at a millisecond-by-millisecond resolution. Illustrated is one such model in which excitatory visual and auditory streams (depicted at three time points) are integrated in real-time by a model neuron, which also receives input from inhibitory sources.

Figure 6.

SC multisensory integration develops gradually in postnatal life. Top, Multisensory enhancement in SC neurons of cat and monkey are not present in early neonatal life. The visual-auditory (VA) response in the exemplar neurons is not significantly better than the V response. Bottom, Normal adult exemplars. Bar graph conventions are the same as Fig. 2.

Figure 7.

Lack of multisensory experience compromises multisensory development, but explicit training can compensate. Left, Experimental manipulations to preclude visual-auditory experience include dark-rearing (top) and rearing with masking noise (bottom). Both rearing conditions disrupt the development of the ability to integrate those cross-modal stimuli. Multisensory enhancement (ME) is not significantly above zero. Right, Explicit training with spatiotemporally concordant visual-auditory cues can mitigate these deficits, even in adulthood.

Figure 8.

Multisensory training can restore visual function in cortically blinded animals. Lesions of all contiguous regions of visual cortex on one side (e.g., left) of the cat brain (shaded area on the schematic) result in complete blindness in contralesional (right) space. Visual responses are now restricted to left visual space (the proportion of correct responses at each location is shown in green on the polar plots). But after repeated exposure to congruent visual-auditory stimuli in the blinded visual field, vision is restored there.