Changing the Tendency to Integrate the Senses

Abstract

Integration of sensory signals that emanate from the same source, such as the visual of lip articulations and the sound of the voice of a speaking individual, can improve perception of the source signal (e.g., speech). Because momentary sensory inputs are typically corrupted with internal and external noise, there is almost always a discrepancy between the inputs, facing the perceptual system with the problem of determining whether the two signals were caused by the same source or different sources. Thus, whether or not multisensory stimuli are integrated and the degree to which they are bound is influenced by factors such as the prior expectation of a common source. We refer to this factor as the tendency to bind stimuli, or for short, binding tendency. In theory, the tendency to bind sensory stimuli can be learned by experience through the acquisition of the probabilities of the co-occurrence of the stimuli. It can also be influenced by cognitive knowledge of the environment. The binding tendency varies across individuals and can also vary within an individual over time. Here, we review the studies that have investigated the plasticity of binding tendency. We discuss the protocols that have been reported to produce changes in binding tendency, the candidate learning mechanisms involved in this process, the possible neural correlates of binding tendency, and outstanding questions pertaining to binding tendency and its plasticity. We conclude by proposing directions for future research and argue that understanding mechanisms and recipes for increasing binding tendency can have important clinical and translational applications for populations or individuals with a deficiency in multisensory integration.

Keywords: binding tendency; coupling prior; crossmodal binding; integration learning; integration tendency; multisensory binding; multisensory integration; multisensory learning; multisensory plasticity; probability of common cause; sensory binding.

Figure 1

The generative model of Bayesian causal inference [5]. Two causal structures can give rise to auditory and visual stimuli $x_A$ and $x_V$ . Either a common cause ( $s)$ has produced the stimuli $\left(C=1\right)$ , or independent causes $s_A$ and $s_{\mathrm{V}}$ , have produced the two stimuli $\left(C=2\right)$ .

Figure 2

Learning mechanisms involved in binding tendency plasticity. Two possible learning mechanisms underlying the plasticity of binding tendency are illustrated. (a) The statistical learning mechanism: In this mechanism, every sensory event is informative and influential towards the tendency to bind. If an observer is frequently exposed to congruent audiovisual stimuli, each exposure increases the prior probability of a common cause by offering evidence of increased frequency of common-cause audiovisual stimuli. If an observer is frequently exposed to incongruent audiovisual stimuli, each exposure decreases the prior probability of a common cause. (b) The predictive coding mechanism. In this mechanism, only sensory events that contradict the internal generative model of the sensory environment will elicit updates in the prior probability of a common cause. For instance, if the internal generative model of the world expects that incongruent stimuli do not have a common cause, and an observer repeatedly experiences spatially incongruent but temporally congruent stimuli, these repeated sensory experiences contradict the internal model of the world and produce a model prediction-error signal. This results in a relaxation of binding tendency and an increased probability of common cause to account for the congruent stimuli that are nonetheless spatially discrepant. Adapted with permission from [28].

Figure 3

Possible neural substrates of binding tendency. The learned or hardwired association between the senses may be mediated by the lateral connectivity between unisensory regions, here denoted by the red arrow between auditory and visual regions. The strength of this connection would map to the degree of binding tendency. Extant models of multisensory integration have utilized this mechanism (see [45] as an example). Top-down influences on binding tendency may be mediated by projections from cognitive processing regions in the frontal lobe to multisensory processing areas involved in causal inference. This is denoted by the green arrow.