Capture of auditory motion by vision is represented by an activation shift from auditory to visual motion cortex

Abstract

The brain is capable of integrating motion information arising from visual and auditory input. Such integration between sensory modalities can aid one another and helps to stabilize the motion percept. However, if motion information differs between sensory modalities, it can also result in an illusory auditory motion percept. This phenomenon is referred to as the cross-modal dynamic capture (CDC) illusion. We used functional magnetic resonance imaging to investigate whether early visual and auditory motion areas are involved in the generation of this illusion. Among the trials containing conflicting audiovisual motion, we compared the trials in which CDC occurred to those in which it did not and used a region of interest approach to see whether the auditory motion complex (AMC) and the visual motion area hMT/V5+ were affected by this illusion. Our results show that CDC reduces activation in bilateral auditory motion areas while increasing activity in the bilateral hMT/V5+. Interestingly, our data show that the CDC illusion is preceded by an enhanced activation that is most dominantly present in the ventral intraparietal sulcus. Moreover, we assessed the effect of motion coherency, which was found to enhance activation in bilateral hMT/V5+ as well as in an area adjacent to the right AMC. Together, our results show that audiovisual integration occurs in early motion areas. Furthermore, it seems that the cognitive state of subjects before stimulus onset plays an important role in the generation of multisensory illusions.

Figure 1.

A graphical representation of the audiovisual stimuli used in the experiment. Stimulation consisted of continuous visual motion and auditory apparent motion. The spheres represent the location of the visual stimuli, and the speakers represent the perceived location of the auditory stimuli over time during a trial. Auditory stimuli were presented with a spatial resolution of 5°. Stimuli locations are depicted for the 20 intervals during which an auditory stimulus was presented for 100 ms (interstimulus interval was 50 ms). An example is shown of a coherent trial in which audiovisual stimuli move coherently (top) and of a conflicting trial (bottom) in which motion direction was opposite across senses. During the experiment, these trials were presented together with an equal amount of their mirrored equivalents.

Figure 2.

Left, The group average of the behavioral data showing the amount of misclassification during the different conditions (including SEs across subjects). During all types of trials in which auditory stimuli were presented, subjects reported the direction of auditory motion, whereas during the visual trials subjects responded to the direction of visual motion. Right, The misclassification rate for each single subject.

Figure 3.

Significantly activated regions displayed on a map representing the flattened cortex of the left and right hemisphere of subject IEN24. Areas are displayed that show significant effects (p < 0.05 corrected) for the following contrasts: coherent > conflicting (green), conflicting > coherent (red), conflicting > cross-modal dynamic capture (orange), and cross-modal dynamic capture > conflicting (purple). The flat map further includes areas that show significant motion sensitivity (p < 0.005 corrected) for visual motion and auditory motion using the contrasts flowfield RDP > static RDP for visual motion (yellow) and rotating sound > static sound for auditory motion (blue). CS, Central sulcus; dIPS, dorsal intraparietal sulcus; IFS, inferior frontal sulcus; ITS, inferior temporal sulcus; LS, lateral sulcus; poCS, postcentral sulcus; prCS, precentral sulcus; SFS, superior frontal sulcus; STS, superior temporal sulcus; vIPS, ventral intraparietal sulcus; V1d–V3d, dorsal visual areas 1–3; V1v–V3v, ventral visual areas 1–3.

Figure 4.

Left, The grand mean event-related responses generated using deconvolution for the ROIs hMT/V5+ and the AMC as defined in the localizer experiment. The ROI for the hMT/V5+ was defined on an individual level and the AMC on a group level. Event-related responses are shown for coherent audiovisual trials (green), conflicting audiovisual trials with correct responses (red), and conflicting audiovisual trials with wrong responses, i.e., inducing the illusion of cross-modal dynamic capture (brown), auditory (blue), and visual (yellow) trials. Right, Individual differences of the peak of the BOLD response (mean of data points 3–8) between coherent and conflicting trials (green) and the difference between cross-modal dynamic capture and conflicting trials (brown). Data points were recorded each second, the first starting at stimulus onset.