Factors influencing audiovisual fission and fusion illusions

Abstract

Information processing in auditory and visual modalities interacts in many circumstances. Spatially and temporally coincident acoustic and visual information are often bound together to form multisensory percepts [B.E. Stein, M.A. Meredith, The Merging of the Senses, A Bradford Book, Cambridge, MA, (1993), 211 pp.; Psychol. Bull. 88 (1980) 638]. Shams et al. recently reported a multisensory fission illusion where a single flash is perceived as two flashes when two rapid tone beeps are presented concurrently [Nature 408 (2000) 788; Cogn. Brain Res. 14 (2002) 147]. The absence of a fusion illusion, where two flashes would fuse to one when accompanied by one beep, indicated a perceptual rather than cognitive nature of the illusion. Here we report both fusion and fission illusions using stimuli very similar to those used by Shams et al. By instructing subjects to count beeps rather than flashes and decreasing the sound intensity to near threshold, we also created a corresponding visually induced auditory illusion. We discuss our results in light of four hypotheses of multisensory integration, each advocating a condition for modality dominance. According to the discontinuity hypothesis [Cogn. Brain Res. 14 (2002) 147], the modality in which stimulation is discontinuous dominates. The modality appropriateness hypothesis [Psychol. Bull. 88 (1980) 638] states that the modality more appropriate for the task at hand dominates. The information reliability hypothesis [J.-L. Schwartz, J. Robert-Ribes, P. Escudier, Ten years after Summerfield: a taxonomy of models for audio-visual fusion in speech perception. In: R. Campbell (Ed.), Hearing by Eye: The Psychology of Lipreading, Lawrence Earlbaum Associates, Hove, UK, (1998), pp. 3-51] claims that the modality providing more reliable information dominates. In strong forms, none of these three hypotheses applies to our data. We re-state the hypotheses in weak forms so that discontinuity, modality appropriateness and information reliability are factors which increase a modality's tendency to dominate. All these factors are important in explaining our data. Finally, we interpret the effect of instructions in light of the directed attention hypothesis which states that the attended modality is dominant [Psychol. Bull. 88 (1980) 638].