Development of multisensory integration from the perspective of the individual neuron

Abstract

The ability to use cues from multiple senses in concert is a fundamental aspect of brain function. It maximizes the brain’s use of the information available to it at any given moment and enhances the physiological salience of external events. Because each sense conveys a unique perspective of the external world, synthesizing information across senses affords computational benefits that cannot otherwise be achieved. Multisensory integration not only has substantial survival value but can also create unique experiences that emerge when signals from different sensory channels are bound together. However, neurons in a newborn’s brain are not capable of multisensory integration, and studies in the midbrain have shown that the development of this process is not predetermined. Rather, its emergence and maturation critically depend on cross-modal experiences that alter the underlying neural circuit in such a way that optimizes multisensory integrative capabilities for the environment in which the animal will function.

Figure 1. The organization and development of the multisensory superior colliculus

a | The cut-away diagram shows the location of the superior colliculus (SC) in the midbrain of the cat and the association cortex (anterior ectosylvian sulcus (AES) and rostrolateral suprasylvian sulcus (rLS)), from which the SC receives crucial cortico-collicular inputs. b | The three sensory representations (visual, auditory and somatosensory; shown at the top) in the SC are organized into an overlapping multisensory topographic map, as shown below (grey map). In each individual map, the horizontal meridian runs roughly rostral–caudal and the vertical meridian runs medial– lateral. Thus, forward or central space is represented rostrally, rearward or peripheral space is represented caudally, superior space is represented medially and inferior space is represented laterally. The multisensory map shows the topographic correspondence among the three maps, with the purple regions encompassing the variations in the two meridians that exist among the three maps. External events, such as the presence of the rodent, are often registered by multiple senses (in this case, vision and audition) and relayed via converging cross-modal afferents onto common multisensory target neurons in the map, which are exemplified by crosses in the maps. In adult animals, this leads to enhancements in neuronal activity (that is, physiological salience) and, behaviourally, to a higher probability of detecting the event, localizing in space and orienting to it. c | The basic developmental chronology of sensory responsiveness within the deep layers of the cat SC is shown. Some neurons are already responsive to touch (somatosensation) prenatally. Hearing (audition) becomes effective in activating some SC neurons before the end of the first week of age and sight (vision) at approximately 3 weeks. Despite the convergence of inputs that produces multisensory neurons early in life, these neonatal multisensory neurons cannot yet integrate their cross-modal inputs. This capability for multisensory integration does not appear until approximately 4 weeks of age and gradually matures until the adult-like condition is achieved after several months. Part a is adapted with permission from REF. 101, The American Physiological Society. Part b is adapted with permission from Stein, Barry E., and M. Alex Meredith., The Merging of the Senses, Figure 8.1, © 1993 Massachusetts Institute of Technology, by permission of The MIT Press.

Figure 2. Developmental profile of multisensory integration in the cat superior colliculus

Multisensory neurons (shown in red) first appear in the second postnatal week and steadily increase in number thereafter, nearing adult levels by postnatal week 20. However, the first neurons with multisensory integration capabilities (shown in green) are not seen until the fourth week of life. Thereafter, their incidence increases roughly in parallel with the total incidence of multisensory neurons. The multisensory responses of neurons with multisensory integration capabilities are nearly always depressed by deactivating the association cortex, from which descending cortico-collicular afferents originate (shown in blue). Although the timing of these three developmental trajectories is parallel, the delay in the development of multisensory integration is consistent with the idea that the ability to respond to multiple modalities and the ability to integrate the information they provide are different phenomena, which are mediated by related but not identical developmental processes. Inset bar graphs provide sample responses from individual multisensory neurons at three different ages, one before the development of multisensory integration capabilities (left), one after this development (middle) and one from an adult animal. The responses from the adult neuron are also displayed as ‘impulse rasters’ in which each dot represents a single impulse and each row (ordered from bottom to top) represents the response to a single stimulus presentation. The grey bars show multisensory enhancement (ME). A, auditory; S, somatosensory; V, visual; VA, visual–auditory; VS, visual–somatosensory. Republished with permission of Society for Neuroscience, from Development of multisensory neurons and multisensory integration in cat superior colliculus. Wallace, M. T. & Stein, B. E. 17, 1997; permission conveyed through Copyright Clearance Center, Inc.

Figure 3. A synergy between unisensory subregions of the association cortex drives multisensory integration capabilities in the mature superior colliculus

Depicted is the degree of multisensory enhancement (ME) above the largest unisensory response in a typical visual–auditory superior colliculus neuron when the auditory subregion (the auditory field of the anterior ectosylvian sulcus (FAES)) and/or visual subregion (anterior ectosylvian visual area (AEV)) of the AES was reversibly deactivated. In the control condition, the average multisensory response exceeded the largest average unisensory component response (shown as 100% response on the graph). However, deactivation of either cortical subregion alone eliminated this multisensory enhancement, rendering the multisensory response statistically indistinguishable (denoted by NS) from the largest unisensory response. Subsequent reactivation after each deactivation series restored the neuron’s functional capabilities. Asterisks indicate statistical significance.

Figure 4. The development of multisensory integration depends on concordant experience with cross-modal cues

Depicted are the responses of exemplar neurons that illustrate the most common outcomes of four different rearing conditions. Shown for each exemplar are summary histograms describing the visual (V), auditory (A) and multisensory (VA) responses as the average number of impulses elicited by each stimulus. Also indicated (dashed line) is the sum of the average V and A responses in each condition, as well as the percentage increase elicited by their combined presentation (horizontal lines above each bar indicate the SEM, NS indicates not significantly different from the greatest unisensory response in that condition and asterisks indicate a statistically significant difference). Rearing with visual experience but with degraded auditory experience (that is, noise rearing, left), with auditory experience but without visual experience (that is, dark rearing, right) or with random independent visual and auditory experience (bottom) yields multisensory superior colliculus (SC) neurons lacking multisensory integration capabilities, as shown by the lack of significant difference between the bars representing the multisensory (VA) response and the strongest unisensory (V or A) response. However, rearing with concordant VA experience (top middle) allows SC neurons to develop their multisensory integration capabilities, as shown by the significant increase in the VA response, which in many cases exceeded even the sum of a neuron’s unisensory responses. Republished with permission of Society for Neuroscience, from Incorporating cross-modal statistics in the development and maintenance of multisensory integration. Xu, J., Yu, L., Rowland, B. A., Stanford, T. R. & Stein, B. E. 32, 2012; permission conveyed through Copyright Clearance Center, Inc.

Figure 5. Deactivating the association cortex in early life delays the development of multisensory integration

Unilateral muscimol-infused implants were used to deactivate neurons in the association cortex during the period in early life when superior colliculus (SC) multisensory integration capabilities are first being instantiated. During this time period (not shown), these cortical neurons were unable to process (contralateral) sensory information and were unable to influence their ipsilateral SC target neurons (which also respond to contralateral stimuli). a | When the animals had matured to 1.5 years of age, they were tested on their ability to locate events in space. Their performance was significantly impaired. They were unable to show the normal enhanced localization ability to events in contralateral space that had both visual and auditory components. However, these performance benefits were normal when the events were in ipsilateral space. This behavioural deficit was paralleled by a physiological deficit in ipsilateral SC neurons (left inset). Most failed to produce a better response to the visual–auditory combination than to the most effective of these stimuli individually (multisensory enhancement (ME)). b | However, when animals were re-tested on the same task at 4 years of age, behavioural performance on both sides of space was equivalent, and the SC physiological deficits seemed to have been resolved as well (inset). Apparently, the circuit was still able to acquire the experience needed to develop its multisensory integration capability during adulthood (albeit much more slowly).

Figure 6. Adult plasticity in multisensory integration

Illustrated are three conditions in which multisensory (visual–auditory (VA)) plasticity has been shown. a | A raster display (each dot represents one impulse and each row represents the response to a single stimulus presentation, ordered bottom to top) shows the effect of repeated presentations of sequentially arranged spatially concordant VA stimuli on a multisensory superior colliculus (SC) neuron. Square waves atop the display represent the stimuli. Their repetition increased the magnitude and duration of this characteristic neuron’s response to the first stimulus in the sequence and decreased the latency of the response to the second, leading to the minimization of the temporal gap between the response trains. The changes can be seen by comparing the first set of trials in the grey zone at the bottom of the raster, with the last set of trials, also in grey, at the top. b | The visual and auditory receptive fields of an exemplar SC neuron are shown on the left on a polar plot of VA space (each concentric circle is 10 degrees). Repeated presentation of spatiotemporally concordant VA stimuli increased both the multisensory and unisensory responses of this characteristic neuron. The results of preliminary tests illustrated on the pre-exposure graph show that there was no auditory response (A) and the average multisensory response (VA) was 76% greater than the largest unisensory (V) response (and thus greater than their sum (dashed line)). After repeated exposure to the VA stimulus, magnitudes of all responses (post-exposure graph) were enhanced, and the previously subthreshold (auditory) input was ‘exposed’. c | Multisensory neurons in adult dark-reared animals initially do not integrate cross-modal stimuli but can be rapidly trained to do so by repeated exposure to spatiotemporally concordant VA stimuli. The visual and auditory receptive fields of three exemplar neurons are shown on the left on a polar plot of VA space (each concentric circle is 10 degrees). At the bottom are the numbers of cross-modal exposures provided to each (3,600–50,400). In the middle are the raster displays in response to V, A and AV stimuli, and to the right are graphs of the average impulse counts in each condition. The receptive fields retain immature (that is, large) sizes (compared with that shown in part b for example) with this impoverished sensory experience despite developing their integrative capability, and the magnitude of the integrated multisensory response is larger in neurons with more cross-modal experience (exposure). The horizontal line above each bar represents the SEM. Asterisks indicate statistical significance. Part a is republished with permission of Society for Neuroscience, from Adult plasticity in multisensory neurons: short-term experience-dependent changes in the superior colliculus. Yu, L., Stein, B. E. & Rowland, B. A. 29, 2009; permission conveyed through Copyright Clearance Center, Inc. Part b is reprinted with permission from REF. 145, The American Physiological Society. Part c is republished with permission of Society for Neuroscience, from Initiating the development of multisensory integration by manipulating sensory experience. Yu, L., Rowland, B. A. & Stein, B. E. 30, 2010; permission conveyed through Copyright Clearance Center, Inc.

Figure 7. Learning to integrate requires a neuron to experience both cues simultaneously

Shown on the left are schematics of the receptive fields (RFs) of three visual–auditory (VA) neurons. The visual (icon of a light bulb) and auditory (icon of a speaker) stimuli fell into one or both RFs of a given neuron and were repeatedly presented in close temporal proximity. As shown in the summary histograms to the right, only the neuron in which both stimuli were in their respective RFs ultimately developed the capability to integrate those inputs and showed a significantly enhanced multisensory response (indicated by the asterisks). The horizontal line above each bar represents the SEM. NS, not statistically significant. Republished with permission of Society for Neuroscience, from Initiating the development of multisensory integration by manipulating sensory experience. Yu, L., Rowland, B. A. & Stein, B. E. 30, 2010; permission conveyed through Copyright Clearance Center, Inc.