Multisensory Processes: A Balancing Act across the Lifespan

Abstract

Multisensory processes are fundamental in scaffolding perception, cognition, learning, and behavior. How and when stimuli from different sensory modalities are integrated rather than treated as separate entities is poorly understood. We review how the relative reliance on stimulus characteristics versus learned associations dynamically shapes multisensory processes. We illustrate the dynamism in multisensory function across two timescales: one long term that operates across the lifespan and one short term that operates during the learning of new multisensory relations. In addition, we highlight the importance of task contingencies. We conclude that these highly dynamic multisensory processes, based on the relative weighting of stimulus characteristics and learned associations, provide both stability and flexibility to brain functions over a wide range of temporal scales.

Keywords: aging.; cross-modal; development; learning; multisensory; plasticity.

Figure 1

Schema of cortical loci of multisensory processes. The schema is depicted on a left hemisphere, with the occipital lobe on the right side of the image and the frontal lobe on the left side. Primary visual (V1), auditory (A1), and somatosensory (S1) cortices are indicated by the red, green, and blue blobs, respectively. The correspondingly coloured arrows depict a schema where interactions are restricted to higher-level association cortices, such as the prefrontal cortices (PFC) and superior temporal sulcus (STS) (indicated by white disks). The black lines depict a schema where interactions occur directly between low-level cortices. There is now evidence in support of both schemas. Therefore, multisensory processes undoubtedly involve a dynamic combination of these schemas, which emerge as a consequence of both sensory stimulus- and experience-dependent processes.

Figure 2

Schema and examples of the interplay between sensitivity to physical stimulus characteristics and sensitivity to learned associations between stimuli in shaping multisensory functions across various time scales. Sensitivity to physical stimulus characteristics is represented in blue, and sensitivity to learned associations is represented in red. (A) The interplay between these factors takes place over multiple time scales, ranging from seconds to years (i.e., lifespan). Task contingencies can influence the relative weight attributed to a given factor. (B) Illustration of the shift in the sensitivity from a heavy early dependence on stimulus timing (i.e., synchrony) to a greater dependence on stimulus identity at later time periods. Boxes reflect the “binding” process and the shift from low-level temporal factors to higher-level learned associations. In some cases, including infancy, this shift is itself linked with perceptual narrowing, as depicted in panel (C). (C) Illustration of perceptual narrowing characterized by early broad multisensory tuning and later narrower tuning. The broad tuning leads infants to bind non-native as well as native auditory and visual stimuli largely because responsiveness is mediated by low-level synchrony cues, whereas the later narrower tuning leads infants to only bind native auditory and visual stimuli because responsiveness is now mediated by higher-level identity cues. (D) An example of how training can modify multisensory processes. Here the left and right panels represent the audiovisual temporal binding window as measured by synchrony judgments before and after feedback training. Note the post-training narrowing of the distribution, reflective of the malleability in multisensory temporal acuity. Importantly, it should be noted that these training effects must be interpreted in the larger developmental context where plasticity generally decreases from infancy into adulthood. Thus, these results indicate that there is a great deal of latent plasticity in adulthood. (E) An example of how memory function (e.g. assayed by repetition discrimination with unisensory visual stimuli) is affected when prior stimuli are presented in a multisensory context. When this context entails meaningless sounds such as an image of a duck paired with a pure tone, recall accuracy is impaired versus that for images only presented visually. By contrast, when the prior context entails semantically congruent meaningful sounds, such as an image of a duck paired with a quacking sound, recall accuracy is improved versus that for images only presented visually. (F) An example of how training with a sensory substitution device can modify responses within higher-level visual cortices (here the vWFA). Prior to training, differential responses are observed only during Braille-reading of words, but not when hearing transformations of written words via a sensory substitution device. After training, differential responses are observed for both Braille-reading and sensory substitution.

Figure 3

Schemas of the consequences for multisensory integration across three canonical developmental stages. In these schemas, auditory and visual stimulus parameters are denoted by red and blue geometric shapes, respectively, and corresponding shapes refer to features of the same object. The curves in these schemas refer to tuning profiles of neural populations, and the tuning function for an exemplar stimulus parameter (i.e., ‘square’) is highlighted. The right side of these schemas depict putative responses to co-presentation of a given auditory and visual stimulus parameter. (A) During an immature stage, neural tuning is extremely broad and responses are typically of low magnitude (due principally to the immaturity of the sensory systems themselves). Although multisensory convergence is likely, multisensory integration does not occur. (B) During an intermediate stage, neural and representational tuning narrows, but still remains relatively broad. Multisensory interactions are now seen (denoted by purple rectangles), and can occur for a broader range of stimulus attributes than seen at later stages. In these initial two stages, low-level physical stimulus characteristics bootstrap the construction of category-general, multisensory representations that become more specific and constrained due to experience with native multisensory inputs. This occurs particularly extensively over the first year of life in humans. (C) During a final stage, neural and representational tuning narrows and becomes highly specialised as behaviours become increasingly refined and sophisticated. While integration occurs with co-presentation of stimulus attributes shared across the senses (e.g., same location), no integration occurs with the pairing of unshared attributes. While this absence of integration might resemble what is observed during an immature stage, its cause is instead due to the narrowly tuned representations within each sense (i.e. The ‘circle’ falls outside of the tuning for the ‘square’.). Most importantly, in adulthood there can be dynamic shifts between stages (C) and (B) based on learned associations and task contingencies (denoted by the brown box), which can override low-level stimulus factors in order to promote the integration of a broader class of stimulus attributes. The full mechanisms of these shifts, including any impact on neural tuning and tuning for a given attribute, remain underexplored. Likewise, it is important to note that specific circuits (and their associated functions) mature at different rates, which also remains largely underexplored.