Neural substrates of reliability-weighted visual-tactile multisensory integration

Abstract

As sensory systems deteriorate in aging or disease, the brain must relearn the appropriate weights to assign each modality during multisensory integration. Using blood-oxygen level dependent functional magnetic resonance imaging of human subjects, we tested a model for the neural mechanisms of sensory weighting, termed "weighted connections." This model holds that the connection weights between early and late areas vary depending on the reliability of the modality, independent of the level of early sensory cortex activity. When subjects detected viewed and felt touches to the hand, a network of brain areas was active, including visual areas in lateral occipital cortex, somatosensory areas in inferior parietal lobe, and multisensory areas in the intraparietal sulcus (IPS). In agreement with the weighted connection model, the connection weight measured with structural equation modeling between somatosensory cortex and IPS increased for somatosensory-reliable stimuli, and the connection weight between visual cortex and IPS increased for visual-reliable stimuli. This double dissociation of connection strengths was similar to the pattern of behavioral responses during incongruent multisensory stimulation, suggesting that weighted connections may be a neural mechanism for behavioral reliability weighting.

Keywords: BOLD fMRI; area MT; effective connectivity; intraparietal cortex; structural equation modeling; weighted connections.

Figure 1

The visual stimulus. The visual stimulus consisted of a video of an animated probe (triangular shape) approaching the image of a hand. Three frames of the video are shown. (A) Reliable visual stimulus. Dynamic random noise was overlaid on the visual stimulus. During reliable visual stimulation, the dynamic noise was transparent. (B) Unreliable visual stimulus. During unreliable visual stimulation, the dynamic noise was opaque.

Figure 2

Behavioral measures of visual-somatosensory multisensory integration. (A) In the visual condition (Vis, orange), subjects made a touch/no-touch judgment, discriminating between noisy movies of a probe touching or just missing the finger (see Figure 1). In the somatosensory condition (SS, blue) a touch/no-touch judgment was performed on a background vibration delivered to the finger with or without an additional touch. In the congruent multisensory condition (Vis + SS, green) the touch/no-touch judgment was performed on a touch that was both seen and felt, or neither seen nor felt. The error bars show the SEM (n = 21 subjects). (B) In the incongruent multisensory condition, subjects made a touch/no-touch judgment for stimuli which were reliable in one modality but not the other (e.g., probe clearly missed the finger in the video but a barely detectable touch was delivered in the somatosensory modality). The orange bars show the percentage of responses that corresponded to the visual stimulus; the blue bars show the percentage of responses that corresponded to the somatosensory stimulus, collapsed across touch and no-touch conditions. Subjects responses usually matched the stimulus presented in the more reliable modality, with responses corresponding to the visual modality in the visual-reliable condition (left bars) and the somatosensory modality in the somatosensory-reliable condition (right bars).

Figure 3

Summary of fMRI activations. (A) Activation during performance of the multisensory touch detection task shown on an inflated average cortical surface model (left hemisphere, single subject). The orange circle highlights active visual areas in lateral occipital cortex. The blue circle highlights active areas in inferior parietal lobe, the location of secondary somatosensory cortex. The green circle highlights active areas in and around the intraparietal sulcus (IPS). The horizontal dashed white line shows the intraparietal sulcus, vertical dashed white line shows the postcentral sulcus. (B) Group activation map from n = 8 subjects. (C) Time course of the BOLD response in the visual cortex ROI during 20 s stimulation blocks of each experimental condition, averaged across blocks and subjects (black lines show the mean percent signal change, gray lines show + -SEM). (D) Time course of the somatosensory cortex response. (E) Time course of the IPS response.

Figure 4

Response to reliable and unreliable unisensory stimuli. (A) The average BOLD signal change in the visual cortex ROI during 20 s stimulation blocks of unisensory visual-reliable stimulation (left plot) and visual-unreliable unisensory stimulation (right plot). Black line shows mean response, gray lines shows ±SEM (n = 8 subjects). (B) The average BOLD signal change in the somatosensory cortex ROI during unisensory somatosensory-reliable stimulation blocks (left plot) and somatosensory-unreliable stimulation blocks (right plot).

Figure 5

Connection weights during reliable and unreliable stimulation. (A) Connectivity in the multisensory somatosensory-reliable/visual-unreliable condition in an individual subject, viewed on that subject's inflated cortical surface. Colored regions show areas with a significant fMRI response during the localizer scan used to create the regions of interest, with a different color for each region of interest (orange for visual, blue for somatosensory, green for IPS). The numbers adjacent to each arrow show the weights between that pair of ROIs, as derived from the structural equation model. (B) Connectivity in the multisensory somatosensory-unreliable/visual-reliable condition in the same subject. (C) Group data showing connection strengths across subjects during multisensory reliable and unreliable stimulation (n = 8 subjects). The blue bars show the connection strength from somatosensory cortex to the IPS, the orange bars show the connection strength from the visual ROI to the IPS ROI. The solid bar in each pair represents the reliable condition for that modality; the hatched bar in each pair is the unreliable condition.