Modality-specific attention attenuates visual-tactile integration and recalibration effects by reducing prior expectations of a common source for vision and touch

Abstract

At any moment in time, streams of information reach the brain through the different senses. Given this wealth of noisy information, it is essential that we select information of relevance - a function fulfilled by attention - and infer its causal structure to eventually take advantage of redundancies across the senses. Yet, the role of selective attention during causal inference in cross-modal perception is unknown. We tested experimentally whether the distribution of attention across vision and touch enhances cross-modal spatial integration (visual-tactile ventriloquism effect, Expt. 1) and recalibration (visual-tactile ventriloquism aftereffect, Expt. 2) compared to modality-specific attention, and then used causal-inference modeling to isolate the mechanisms behind the attentional modulation. In both experiments, we found stronger effects of vision on touch under distributed than under modality-specific attention. Model comparison confirmed that participants used Bayes-optimal causal inference to localize visual and tactile stimuli presented as part of a visual-tactile stimulus pair, whereas simultaneously collected unity judgments - indicating whether the visual-tactile pair was perceived as spatially-aligned - relied on a sub-optimal heuristic. The best-fitting model revealed that attention modulated sensory and cognitive components of causal inference. First, distributed attention led to an increase of sensory noise compared to selective attention toward one modality. Second, attending to both modalities strengthened the stimulus-independent expectation that the two signals belong together, the prior probability of a common source for vision and touch. Yet, only the increase in the expectation of vision and touch sharing a common source was able to explain the observed enhancement of visual-tactile integration and recalibration effects with distributed attention. In contrast, the change in sensory noise explained only a fraction of the observed enhancements, as its consequences vary with the overall level of noise and stimulus congruency. Increased sensory noise leads to enhanced integration effects for visual-tactile pairs with a large spatial discrepancy, but reduced integration effects for stimuli with a small or no cross-modal discrepancy. In sum, our study indicates a weak a priori association between visual and tactile spatial signals that can be strengthened by distributing attention across both modalities.

Keywords: Attention; Causal inference; Multisensory integration; Recalibration; Ventriloquism; Visual-tactile.

Fig. 12.

Causal-inference models of cue integration. The schematic depicts the generative or world model. Visual and tactile signals either arise from a single source (C = 1) or two sources (C = 2). The locations of the visual (s_v) and tactile (s_t) sources vary randomly from trial to trial. In the original version of the model, the distribution of visual and tactile sources over space is described by one Gaussian distribution (e.g. $s_{vt}~\mathcal{N}\left({\mu }_{p·vt},{\sigma }_{p·vt}^2\right)$). Guided by our experimental data, we tested differently shaped modality-specific distributions of visual and tactile sources over space (“Type of sensory prior”). If the visual and the tactile stimulus arise from the same source C = 1, both stimuli originate at a single random location (s_v = s_t = s_vt); if they arise from separate sources C = 2, the two signals have two independent source locations (s_v,s_t). Finally, the observer receives noisy measurements either distributed around these world locations or shifted in one direction (“Type of likelihood”). Moreover, the level of noise perturbing the measurements can vary with attention (“Sensory reliability”). The ideal-observer model assumes the observer performs Bayesian inference, that is, inverts the generative model and determines the posterior probability of each possible location given the noisy measurements. The ideal observer has knowledge about the structure of the world model as well as about the parameters of this world such as the amount of sensory noise, form of sensory priors, and the probability that visual and tactile signals arise from the same source (“Common-source prior”). The observer can use an optimal or heuristic “decision rule” to turn the posterior probability over space into a response indicating either the location of the response-relevant stimulus or the spatial alignment of both stimuli. Variations in the model elements (printed in black) constitute the different model variants we tested to find the best model description of the specific world scenario and the rules our participants employed. Having established the best-fitting model variant, we tested for attention-dependent variations in several model parameters (printed in red). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Fig. 13.

Simulated ventriloquism effect — localization shifts. Simulated and observed group mean localization responses as a function of the physical stimulus location (negative numbers indicate locations closer to the elbow) split by inter-modal discrepancy (negative numbers indicate that visual stimuli were closer to the elbow than the tactile stimulus; ‘none’ indicates that only one modality was presented). For each participant, we simulated 2000 trials per location using participant-level parameter estimates for the overall best-fitting model. Error bars show standard errors of the mean.

Fig. 14.

Simulated ventriloquism effect — perceived spatial alignment. Simulated and observed group mean proportions of alignment responses as a function of the inter-modal discrepancy (negative numbers indicate that visual stimuli were closer to the elbow than the tactile stimulus). For each participant, we simulated 2000 trials per location using participant-level parameter estimates for the overall best-fitting model. Error bars show standard errors of the mean.

Fig. 1.

Experimental setup. (A) Front view. The non-dominant arm was placed under a translucent cover and supported with a cushion. The dominant hand held a joystick. A projector mounted overhead was used to display task instructions, a fixation cross, a cursor, and colored response cues on the translucent cover. (B) Top view. During the response phase a cursor controlled by the joystick was used to indicate the perceived stimulus location. C) Close up. Seven LEDs and seven tactile stimulators were placed along the top of the forearm (15 mm spacing) using a custom-made sleeve. The LEDs faced upward and the tactile stimulators were in contact with the arm. When activated, the LEDs would shine through the translucent cover and the tactile stimulators would vibrate.

Fig. 2.

Experimental design of Expt. 1. Visual-tactile stimulus pairs with a random inter-modal spatial discrepancy, as well as visual and tactile stimuli presented alone, were interleaved across trials. Participants localized either a visual or a tactile stimulus and indicated afterward whether they perceived the stimulus of the other modality as spatially aligned or misaligned. Alternatively, they could indicate that they had only perceived one stimulus. The cue indicating the response-relevant modality was either given before (modality-specific-attention group) or after (distributed-attention group) the stimulus.

Fig. 3.

Expt. 1: Ventriloquism effect — localization shifts. The ventriloquism effect was measured by subtracting localization responses in trials in which only one stimulus was presented from localization responses for stimuli of the same modality and location but in trials in which a visual-tactile stimulus pair was presented. The averaged differences in perceived location are shown as a function of the spatial discrepancy within the visual-tactile stimulus pair (five possible locations for each modality resulting in nine visual-tactile discrepancies; negative numbers indicate pairs with the visual stimulus located closer to the elbow than the tactile stimulus). The data are separated into trials in which the visual and tactile stimuli were perceived as spatially aligned (open circles and dotted lines) or misaligned (solid circles and lines; see Fig. S1 for undivided data). Visual (top row, orange) and tactile (bottom row, blue) stimuli were response-relevant with equal probability. The modality-specific-attention group (left column) was informed about the response-relevant modality before the stimulation; the distributed-attention group (right column) was informed about the relevant modality after the stimulation. Regression lines are based on group averages of individual participants’ intercept and slope parameters. The grey diagonal line indicates the maximal possible localization shift, the horizontal line indicates the absence of a localization shift, and error bars are standard errors of the mean. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Fig. 4.

Expt. 1: Ventriloquism effect — perceived spatial alignment. Group mean proportion of responses indicating that the stimuli of a visual-tactile stimulus pair were perceived as spatially aligned. Data from trials in which the visual (orange) or tactile (blue) stimulus had to be localized are shown separately for the modality-specific-attention group (left side), which was informed about the response-relevant modality before the stimulation, and the distributed-attention group (right side), which was informed about the relevant modality after the stimulation. Error bars show standard errors of the mean. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Fig. 5.

Design of Expt. 2: Visual recalibration of touch. The experiment had three phases: pre-adaptation, adaptation, and post-adaptation. Pre- and post-adaptation phases: participants localized visual and tactile stimuli presented alone. Adaptation phase: visual-tactile stimulus pairs were presented with a fixed spatial discrepancy. The visual stimulus was always located 30mm closer to the elbow than the tactile stimulus. The task during the adaptation phase differed between groups. Non-spatial attention group: participants detected occasional stimulus pairs with a longer duration.Spatial attention groups: participants localized either the visual or the tactile stimulus in the visual-tactile stimulus pair. The localization tasks differed with respect to the time point at which the response-relevant modality was cued, the cue was presented either before (modality-specific attention) or after (distributed attention) the stimulus.

Fig. 6.

Expt. 2: Visual and tactile localization performance in the pre- and post-adaptation phases. Group mean localization responses are shown as a function of the physical stimulus location for visual (top row, orange hues) and tactile stimuli (bottom row, blue hues). Negative numbers indicate locations closer to the elbow. Localization responses are shown for the pre- (light shades) and post-adaptation (dark shades) phases. Lines are based on group averages of individual participants’ regression intercept and slope parameters. The unity line indicates perfect localization. Error bars show standard errors of the mean. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Fig. 7.

Expt. 2: Ventriloquism aftereffect. Group mean adaptation-induced shifts in the perceived location of visual (orange) and tactile (blue) stimuli. The shifts are calculated as the difference between the pre- and post-adaptation regression intercepts from Fig. 6. Negative numbers indicate a shift of localization responses toward the elbow. Error bars show standard errors of the mean. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Fig. 8.

Group means of parameter estimates for the best-fitting model. (A) σ_v,VT,att+ and σ_v,VT, indicative of the reliability of attended visual stimuli that were presented with a tactile stimulus. (B) σ_t,VT,att+ and σ_t,VT, indicative of the reliability of attended tactile stimuli that were presented in the context of visual-tactile stimulation. (C) p_CC: participants’ a priori probability that both stimuli share a common source. Error bars show standard errors of the mean. See Tables S1 and S2 for single-participant parameter estimates

Fig. 9.

Simulations of the influence of sensory reliabilities and common-cause priors on the ventriloquism effect — localization shifts. Simulated shifts in the perceived locations of (A) visual and (B) tactile stimuli presented as part of a visual-tactile stimulus pair with varying degree of cross-modal spatial discrepancy (columns; negative numbers indicate pairs with the visual stimulus located closer to the elbow than the tactile stimulus). The intensity and direction of the perceptual shifts (color scale; negative numbers indicate a shift toward the elbow) are shown as a function of the standard deviation of visual (σ_v) and tactile (σ_t) sensory noise and the prior probability of visual and tactile signals sharing a source (p_CC, rows). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Fig. 10.

Simulations of the influence of sensory reliabilities and common-cause priors on the tactile ventriloquism effect — perceived spatial alignment. Simulated probabilities of perceiving visual-tactile stimulus pairs with varying degree of cross-modal spatial discrepancy (columns; negative numbers indicate pairs with the visual stimulus located closer to the elbow than the tactile stimulus) as originating from a common source. Probabilities (color scale) are shown as a function of the standard deviation of visual (σ_v) and tactile (σ_t) sensory noise and the prior probability of visual and tactile signals sharing a source (p_CC, rows). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Fig. 11.

Simulated tactile ventriloquism aftereffect. Estimated shifts in the localization of tactile stimuli after recalibration (color scale) are shown as a function of participants’ estimates of the probability of visual and tactile signals sharing a source (p_CC) and the standard deviation of the (attended) tactile stimulus (σ_t,VT for the distributed-attention group, σ_t,VT,att+ for the modality-specific-attention group) in visual-tactile trials during the adaptation phase. We additionally varied the decrease in tactile reliability (i.e., the increase in tactile standard deviation, σ_t,att− − σ_t,att+) associated with attention toward the visual stimulus and the learning rate α. (A,B) Results from the optimal Bayesian causal-inference model of adaptation (Sato et al., 2007), for (A) the distributed- and (B) the modality-specific-attention group. (C, D) Results from a model variant in which recalibration only occurs in trials in which the two sensory signals were judged as sharing a source. (E, F) Results from a model variant in which the learning rate is modulated by the posterior probability of a common cause. The red markers indicate the parameter estimates derived for visual-tactile integration (squares: distributed-attention group, circles: modality-specific-attention group). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)