Hearing what the eyes see: auditory encoding of visual temporal sequences

Abstract

When the senses deliver conflicting information, vision dominates spatial processing, and audition dominates temporal processing. We asked whether this sensory specialization results in cross-modal encoding of unisensory input into the task-appropriate modality. Specifically, we investigated whether visually portrayed temporal structure receives automatic, obligatory encoding in the auditory domain. In three experiments, observers judged whether the changes in two successive visual sequences followed the same or different rhythms. We assessed temporal representations by measuring the extent to which both task-irrelevant auditory information and task-irrelevant visual information interfered with rhythm discrimination. Incongruent auditory information significantly disrupted task performance, particularly when presented during encoding; by contrast, varying the nature of the rhythm-depicting visual changes had minimal impact on performance. Evidently, the perceptual system automatically and obligatorily abstracts temporal structure from its visual form and represents this structure using an auditory code, resulting in the experience of "hearing visual rhythms."

Fig. 1

Schematic diagram of the visual and auditory stimuli used in Experiment 1: (a) two visual sequences constituting a same trial accompanied by congruent auditory sequences and (b) two visual sequences constituting a different trial accompanied by two different incongruent auditory sequences. The arrows indicate the timing of the visual beats (points in time when the Gabor patches underwent contrast reversals), and the speakers indicate the timing of the auditory clicks. Note that on both same and different trials, the auditory signals could be either congruent with the visual beats, incongruent with the visual beats, or absent altogether; the two factors were orthogonal, meaning that auditory sequences alone provided no cue for reliable discrimination of same versus different visual sequences.

Fig. 2

Results of Experiment 1: proportion of correct responses as a function of the nature of the auditory sequences. Error bars depict standard errors across observers.

Fig. 3

Results of Experiment 2: proportion of correct responses as a function of the presence or absence of task-irrelevant information in the auditory and visual domains. Error bars depict standard errors across observers.

Fig. 4

Results of Experiment 3: proportion of correct responses as a function of the presence or absence of auditory stimulation during the first (encoding) and second (retrieval) visual sequences. Error bars depict standard errors across observers.