The fusion of mental imagery and sensation in the temporal association cortex

Abstract

It is well understood that the brain integrates information that is provided to our different senses to generate a coherent multisensory percept of the world around us (Stein and Stanford, 2008), but how does the brain handle concurrent sensory information from our mind and the external world? Recent behavioral experiments have found that mental imagery--the internal representation of sensory stimuli in one's mind--can also lead to integrated multisensory perception (Berger and Ehrsson, 2013); however, the neural mechanisms of this process have not yet been explored. Here, using functional magnetic resonance imaging and an adapted version of a well known multisensory illusion (i.e., the ventriloquist illusion; Howard and Templeton, 1966), we investigated the neural basis of mental imagery-induced multisensory perception in humans. We found that simultaneous visual mental imagery and auditory stimulation led to an illusory translocation of auditory stimuli and was associated with increased activity in the left superior temporal sulcus (L. STS), a key site for the integration of real audiovisual stimuli (Beauchamp et al., 2004a, 2010; Driver and Noesselt, 2008; Ghazanfar et al., 2008; Dahl et al., 2009). This imagery-induced ventriloquist illusion was also associated with increased effective connectivity between the L. STS and the auditory cortex. These findings suggest an important role of the temporal association cortex in integrating imagined visual stimuli with real auditory stimuli, and further suggest that connectivity between the STS and auditory cortex plays a modulatory role in spatially localizing auditory stimuli in the presence of imagined visual stimuli.

Keywords: auditory cortex; mental imagery; multisensory; superior temporal sulcus; ventriloquism.

Figure 1.

Task design. Each trial was preceded by 12 s of fixation, followed by instructions and a countdown. The instructions informed the participants that they should imagine the circle on that trial, and the countdown cued the participants to the timing and location (left or right; right in the above example trial) of the to-be-imagined visual stimulus. Following the countdown, the participants imagined the brief appearance (100 ms) of the visual stimulus once per second for 12 s while a brief auditory (100 ms) stimulus was presented in synchrony or asynchrony (i.e., 500 ms following the onset of the first imagined visual stimulus). Participants indicated whether they perceived the sound to come from the left, center, or right of fixation after each 12 s trial.

Figure 2.

Imagery-induced ventriloquism. A, Behavioral results obtained in the scanner revealed a stronger ventriloquist effect when auditory stimuli were presented synchronously with imagined visual stimuli (AV_i sync.) compared with asynchronously (AV_i async.). B, The same effect was found for real visual stimuli presented synchronously with an auditory stimulus (AV sync.) compared with asynchronously (AV async.) during functional localizer scans. Error bars denote ±SEM; asterisks between bars indicate significance (*p's < 0.05).

Figure 3.

Neural basis of imagery-induced ventriloquism. A, Activity associated with audiovisual synchrony (vs asynchrony) for imagined visual stimuli within the functionally defined multisensory regions of interest (fROIs outlined in white) overlaid on a representative inflated cortical surface (left); coronal and sagittal sections displaying the peak activation in the L. STS is overlaid on the average normalized anatomical image from our participants (right). The activity differences observed in the parietal cortex were the result of deactivations, i.e., less activity for synchronous auditory and imagined visual stimuli compared with the resting baseline (see Results). B, Bar plot shows the parameter estimates from the significant peak of activation in the L. STS; error bars denote ±SEM. C, Post hoc multiple-regression analysis demonstrating that the activity in the L. STS in the AV_i synchronous [Av_i sync.; vs AV_i asynchronous (Av_i async.)] condition could be predicted by the strength (i.e., the difference of AV_i synchrony and AV_i asynchrony ventriloquism indices) of the imagery-induced ventriloquist effect. D, Significant enhanced connectivity between the right auditory cortex and the L. STS seed region for the AVi synchronous (vs AVi asynchronous) condition overlaid on a representative inflated cortical surface (bottom left). A yellow circle marks the approximate location of the L. STS seed on an inflated left hemisphere cortical surface (top left; fROIs outlined in white). Coronal and axial sections displaying the peak connectivity to the right auditory cortex are overlaid on the average anatomical image from our participants (right). E, Plot of the PPI for one representative subject showing a steeper regression slope relating L. STS activity to the response magnitude of the right (R.) auditory cortex during the AV_i synchrony (AV_i sync., green) compared with the AV_i asynchrony condition (AV_i async., black). F, Post hoc multiple-regression analysis demonstrating that effective connectivity from L. STS to the right (R.) auditory cortex in the AV_i synchrony (vs AV_i asynchrony) condition could be predicted by the strength of the imagery-induced ventriloquist effect. Activation maps are displayed at p_uncorrected < 0.01 for display purposes only.

Figure 4.

Effective connectivity during perceptual functional localizer scans. A, Significant enhanced connectivity between the left auditory cortex (−51, −27, 6; t₍₂₁₎ = 3.74, p_{FWE-corrected} < 0.05) and the L. STS seed region for the AV synchrony (vs AV asynchrony) condition overlaid on a representative inflated cortical surface (left) and in a coronal (top right) and axial (bottom right) section of the average anatomical image from our participants. A yellow circle marks the approximate location of the L. STS seed. B, Plot of the PPI for one representative subject showing a steeper regression slope relating L. STS activity with the response magnitude of the left (L.) auditory cortex during the AV synchrony (AV sync., blue) compared with the AV asynchrony condition (AV async., black). C, Significant (63, −25, 9; t₍₂₁₎ = 3.65, p_{FWE-corrected} < 0.05) enhanced connectivity between the right (R.) auditory cortex and the L. STS seed region for the AV synchrony (vs AV asynchrony) condition overlaid on a representative inflated cortical surface (left) and in the coronal (top right) and axial (bottom right) section of the average anatomical image from our participants. D, Plot of the PPI for one representative subject showing a steeper regression slope relating L. STS activity with the response magnitude of the right (R.) auditory cortex during the AV synchrony (AV sync., blue) compared with the AV asynchrony condition (AV async., black). Activation maps are displayed at p_uncorrected < 0.005 for display purposes.