Embodied Cross-Modal Interactions Based on an Altercentric Reference Frame

Abstract

Accurate comprehension of others' thoughts and intentions is crucial for smooth social interactions, wherein understanding their perceptual experiences serves as a fundamental basis for this high-level social cognition. However, previous research has predominantly focused on the visual modality when investigating perceptual processing from others' perspectives, leaving the exploration of multisensory inputs during this process largely unexplored. By incorporating auditory stimuli into visual perspective-taking (VPT) tasks, we have designed a novel experimental paradigm in which the spatial correspondence between visual and auditory stimuli was limited to the altercentric rather than the egocentric reference frame. Overall, we found that when individuals engaged in explicit or implicit VPT to process visual stimuli from an avatar's viewpoint, the concomitantly presented auditory stimuli were also processed within this avatar-centered reference frame, revealing altercentric cross-modal interactions.

Keywords: altercentric reference frame; cross-modal interactions; embodied processing; visual perspective-taking.

Figure 1

(A,D) Experimental procedures. An exemplary trial in the valid condition featuring a “left” auditory cue and a “left” visual target in Experiment 1a (A). In Experiment 1b (D), the color of the fixation cross serves as an indicator of response rules, while the avatar is not presented. (B,E) Experimental conditions. A demonstration of the valid and invalid conditions corresponding to a “left” visual target, from a left avatar’s visual perspective (B), or according to response rules (E). (F) Response rules in Experiment 1b. The mapping between colors and response rules is counterbalanced across participants. (C,G) Experimental results. Participants’ inverse efficiency scores in the two conditions in Experiments 1a (C) and 1b (G). Error bars indicate one standard error. ** p < 0.01. “n.s.” indicates the lack of statistical significance.

Figure 2

(A) Experimental procedure. An exemplary trial in the congruent condition with a “rightward” auditory motion and a “rightward” visual motion. The yellow arrow indicating the direction of coherent moving dots is not presented during the experiment. The grey fan-shaped symbols represent loudness changes in the left (L) and the right (R) channels of the headset. (B) Experimental conditions. A demonstration of the congruent and incongruent conditions corresponding to a “rightward” visual stimulus from a left avatar’s visual perspective. (C,D) Experimental results. The psychophysical curve (C) illustrates the group-level frequency of reporting “rightward” at different coherence levels of visual motion, where positive values indicate rightward motion and negative values indicate leftward motion. The slopes of the psychophysical curves (D) in the two conditions. Error bars indicate one standard error. ** p < 0.01.

Figure 3

(A) Experimental procedure. An exemplary trial in the congruent condition in Experiment 3a, where a “rightward” auditory motion and a “rightward” coherent visual motion are presented. The directions of the coherent moving dots and the random moving dots are indicated by the yellow arrows, which are not presented during the experiment. The loudness changes in the left (L) and right (R) channels of the headset are depicted by the grey fan-shaped symbols. (B,C) Visual scenes in Experiments 3b and 3c. (D–I) Experimental results. Participants’ accuracies in the congruent and incongruent conditions in Experiments 3a (D), 3b (E), and 3c (F). Their accuracies in different visual (upward vs. downward) and auditory (leftward vs. rightward) motion directions in Experiments 3a (G), 3b (H), and 3c (I). Error bars indicate one standard error. * p < 0.05. ** p < 0.01. *** p < 0.001. “n.s.” indicates the lack of statistical significance.