The Ventriloquist Illusion as a Tool to Study Multisensory Processing: An Update

Abstract

Ventriloquism, the illusion that a voice appears to come from the moving mouth of a puppet rather than from the actual speaker, is one of the classic examples of multisensory processing. In the laboratory, this illusion can be reliably induced by presenting simple meaningless audiovisual stimuli with a spatial discrepancy between the auditory and visual components. Typically, the perceived location of the sound source is biased toward the location of the visual stimulus (the ventriloquism effect). The strength of the visual bias reflects the relative reliability of the visual and auditory inputs as well as prior expectations that the two stimuli originated from the same source. In addition to the ventriloquist illusion, exposure to spatially discrepant audiovisual stimuli results in a subsequent recalibration of unisensory auditory localization (the ventriloquism aftereffect). In the past years, the ventriloquism effect and aftereffect have seen a resurgence as an experimental tool to elucidate basic mechanisms of multisensory integration and learning. For example, recent studies have: (a) revealed top-down influences from the reward and motor systems on cross-modal binding; (b) dissociated recalibration processes operating at different time scales; and (c) identified brain networks involved in the neuronal computations underlying multisensory integration and learning. This mini review article provides a brief overview of established experimental paradigms to measure the ventriloquism effect and aftereffect before summarizing these pathbreaking new advancements. Finally, it is pointed out how the ventriloquism effect and aftereffect could be utilized to address some of the current open questions in the field of multisensory research.

Keywords: cross-modal; multisensory; recalibration; space; ventriloquism.

Figure 1

Typical experimental designs to measure the ventriloquism effect and aftereffect. Exemplarily, letters indicate unimodal auditory (A) trials and relative locations of auditory (A) and visual (V) stimuli in bimodal trials. In an actual experiment, absolute stimulus locations typically vary between trials. (A) Ventriloquism effect. Participants have to localize cross-modal stimuli with varying spatial discrepancies. Unisensory localization is assessed in an optional pretest block. Comparison of responses between equivalent left- and right-side discrepancies or between bimodal and unimodal stimuli reveal the size of the ventriloquism effect. (B) Immediate ventriloquism aftereffect. Intermixed presentation of bimodal and unimodal trials. Localization in unimodal trials is modulated by the cross-modal discrepancy in the directly preceding bimodal trial. (C) Cumulative ventriloquism aftereffect. Unisensory sound localization is measured before and after exposure to cross-modal stimuli with a consistent spatial discrepancy. (D) Design used in Bruns and Röder (2015) to measure the immediate and cumulative ventriloquism aftereffects concurrently. Tones of two different sound-frequencies (A₁ and A₂) are consistently paired with opposite directions of cross-modal spatial discrepancy. Differences in localization responses between unimodal trials preceded by audiovisual trials with leftward vs. rightward discrepancy reveal the immediate aftereffect, and differences between unisensory localization of A₁ vs. A₂ reveal the cumulative aftereffect (see text for details).