Visual Mislocalization of Moving Objects in an Audiovisual Event

Abstract

The present study investigated the influence of an auditory tone on the localization of visual objects in the stream/bounce display (SBD). In this display, two identical visual objects move toward each other, overlap, and then return to their original positions. These objects can be perceived as either streaming through or bouncing off each other. In this study, the closest distance between object centers on opposing trajectories and tone presentation timing (none, 0 ms, ± 90 ms, and ± 390 ms relative to the instant for the closest distance) were manipulated. Observers were asked to judge whether the two objects overlapped with each other and whether the objects appeared to stream through, bounce off each other, or reverse their direction of motion. A tone presented at or around the instant of the objects' closest distance biased judgments toward "non-overlapping," and observers overestimated the physical distance between objects. A similar bias toward direction change judgments (bounce and reverse, not stream judgments) was also observed, which was always stronger than the non-overlapping bias. Thus, these two types of judgments were not always identical. Moreover, another experiment showed that it was unlikely that this observed mislocalization could be explained by other previously known mislocalization phenomena (i.e., representational momentum, the Fröhlich effect, and a turn-point shift). These findings indicate a new example of crossmodal mislocalization, which can be obtained without temporal offsets between audiovisual stimuli. The mislocalization effect is also specific to a more complex stimulus configuration of objects on opposing trajectories, with a tone that is presented simultaneously. The present study promotes an understanding of relatively complex audiovisual interactions beyond simple one-to-one audiovisual stimuli used in previous studies.

Fig 1. Schematic illustration of stimuli for Experiment 1.

The objects moved from opposite sides toward each other. After 630 ms of motion, objects started to reverse their direction with some degree of overlap and returned to their original positions. The starting points for either object (dashed-line circle) were displaced to manipulate the distance between the object centers, that is, the degree of object overlap in a trial-by-trial manner.

Fig 2. Results of Experiment 1.

The psychometric functions fitted to “non-overlapping” judgment data (a) and directional change judgment data (b) from Subject 8. Below the abscissae, the degree of object overlap corresponds to the distance between object centers. Group mean PSEs for tone conditions in the (c) overlap and (d) directional change judgment tasks. Error bars denote standard errors of the mean (n = 19).

Fig 3. Results of Experiment 2.

Group mean PSEs for tone conditions in the (a) overlap and (b) directional change judgment tasks. Error bars denote standard errors of the mean (n = 11).

Fig 4. Schematic illustration of stimuli used in Experiment 3.

(a) Pre-turn motion trajectory condition: the object moved proximal to the fixation point for 690 ms and then disappeared without the object turning. (b) Post-turn motion trajectory condition: the object appeared proximal to the fixation point, moved away from this location for 690 ms, and then disappeared. (c) Full motion trajectory condition: the object moved proximal to the fixation point for 690 ms, reversed its motion, and returned to the original position.

Fig 5. Results of Experiment 3.

Group mean PSEs for tone conditions in combination with the 3 trajectory conditions. Positive (negative) PSEs indicate that the vanishing point, starting point, or turn-point was localized forward (backward) within the pre-turn trajectory, respectively. Error bars denote standard errors of the mean (n = 12).