Benefits of active listening during 3D sound localization

Abstract

In everyday life, sound localization entails more than just the extraction and processing of auditory cues. When determining sound position in three dimensions, the brain also considers the available visual information (e.g., visual cues to sound position) and resolves perceptual ambiguities through active listening behavior (e.g., spontaneous head movements while listening). Here, we examined to what extent spontaneous head movements improve sound localization in 3D-azimuth, elevation, and depth-by comparing static vs. active listening postures. To this aim, we developed a novel approach to sound localization based on sounds delivered in the environment, brought into alignment thanks to a VR system. Our system proved effective for the delivery of sounds at predetermined and repeatable positions in 3D space, without imposing a physically constrained posture, and with minimal training. In addition, it allowed measuring participant behavior (hand, head and eye position) in real time. We report that active listening improved 3D sound localization, primarily by ameliorating accuracy and variability of responses in azimuth and elevation. The more participants made spontaneous head movements, the better was their 3D sound localization performance. Thus, we provide proof of concept of a novel approach to the study of spatial hearing, with potentials for clinical and industrial applications.

Keywords: Active perception; Head movements; Motion tracking; Spatial hearing; Virtual reality.

Fig. 1

Pre-stimulation alignment of head and eyes. A At the beginning of each trial, the participant was free to move their head (symbolized head direction: blue line) and eyes (symbolized cyclopean eye direction: red line). A* In the HMD display: a bold white cross indicates the actual position of the head, and a thin white cross with a central ball indicates the desired position of the head and eyes, respectively. Two gray arrows flank the bold white cross, showing the participant in which direction to move their head to achieve the desired initial position. B* When the desired head position was achieved, the bold cross turned blue. C* When the desired eye-gaze position was reached, the central ball turned blue. D When all criteria were met (head and eye position straight ahead and speaker within a sphere around the predetermined position), all visual stimulations were removed, D* the scene became dark and the sound was delivered

Fig. 2

Normalization to head-centered coordinates. Bird’s-eye and lateral views of initial head position (in black) and 12 predetermined locations (in gray) for all participants (192 trials each), in VICON reference frame (A–C) and head-centered coordinates (B–D). Variability of predetermined locations averaged across 12 positions for each participant in the two reference frames as a function of coordinates x, y, z (E)

Fig. 3

Actual speaker location with respect to predetermined location, in centimeters. Stimulations delivered to all participants, when the 12 predetermined locations were re-aligned to a single coordinate, centered on the origin of the axes. A top view; B front view; C lateral view; D 3D rendering. Ellipses in the 2D panels represent 95% confidence intervals of the distributions

Fig. 4

Behavioral pointing and effects of static and active listening on sound localization. A Bird’s-eye view of all target positions (black dots) and hand-pointing responses (smaller gray and red circles) for each participant, averaged across trials in a quadrant (i.e., front-left, front-right, back-left, back-right) irrespective of sound distance. Color code is a function of listening condition (black: static listening; red: active listening). B Lateral view of all target positions and responses. Responses for each participant are averaged across (left or right) and distance (near, middle or far). C Lateral view of responses in depth (black box plot: static listening; red box plot: active listening). All participants were included

Fig. 5

Effects of static and active listening on sound localization performances. In azimuth dimension, for the hand absolute error (A) and the hand variable error (B) for each participant as a function of listening condition and antero-posterior position of target sounds. In elevation dimension, for the hand absolute error (C) and the hand variable error (D) for each participant. In depth dimension, for the hand absolute error (E) and the hand variable error (F) for each participant and for the three distance (near, middle and far sound position). Bold horizontal lines indicate the mean for all participants. Asterisks indicate significant differences (* p < 0.05; **p < 0.01; ***p < 0.001)

Fig. 6

Head movements during sound emission in the active listening condition. A Box plot of percentage head movements. Note that two participants were identified as outliers (i.e., they fell outside the 1.5 × interquartile range), made almost no head movements during the active listening condition and were thus excluded from subsequent analyses. B Box plot of mean number of head movements once outliers were removed. C, D Polar histogram showing the distribution of head-movement responses for targets in front (C) and back (D) space. Arrows indicate mean head-movement direction, dashed lines indicate ± 1 SE

Fig. 7.

3D error. Scatterplot of the difference in 3D errors between active and static listening conditions (normalized difference based on static listening performance), as a function of percent head movements. Filled circles indicate participants who improved in active compared to static listening; empty circles indicate participants whose sound localization performance decreased in active compared to static listening