Pitch-elevation and pitch-size cross-modal correspondences do not affect temporal ventriloquism

Abstract

Temporal ventriloquism refers to the shift in the perceived timing of a visual stimulus towards a transient auditory stimulus presented close in time. This effect is demonstrated by greater sensitivity of temporal order judgments of two visual stimuli when a sound is presented before the first visual stimulus and after the second visual stimulus. Recent studies suggest that temporal ventriloquism is affected by cross-modal correspondence between auditory pitch and visual elevation but not by correspondence between pitch and visual size. Here we examined the possibility that these results do not reflect a difference in the effects of different types of cross-modal correspondences on temporal ventriloquism but are rather mediated by shifts in visual-spatial attention triggered by preceding auditory stimuli. In Experiment 1, we replicated the results of previous studies that varied with the type of correspondence. Experiment 2 investigated the effects of the second audiovisual stimuli's asynchrony while the first audiovisual stimuli were synchronized. The results, unlike in Experiment 1, revealed that the magnitude of the temporal ventriloquism effect did not change with the congruency of pitch-elevation correspondence. Experiment 3 also indicated that the asynchrony of the first audiovisual stimuli modulated visual discrimination sensitivity irrespective of temporal ventriloquism. These results suggest that cross-modal correspondences do not affect temporal ventriloquism. Greater visual sensitivity when audiovisual stimuli are congruent with pitch-elevation correspondence may be attributable to shifts in visual attention caused by pitches of the preceding auditory stimulus, which speeds up detection of the first visual stimulus.

Keywords: Cross-modal correspondence; Multisensory integration; Temporal ventriloquism; Timing perception; Visual attention.