Audiovisual integration in depth: multisensory binding and gain as a function of distance

Abstract

The integration of information across sensory modalities is dependent on the spatiotemporal characteristics of the stimuli that are paired. Despite large variation in the distance over which events occur in our environment, relatively little is known regarding how stimulus-observer distance affects multisensory integration. Prior work has suggested that exteroceptive stimuli are integrated over larger temporal intervals in near relative to far space, and that larger multisensory facilitations are evident in far relative to near space. Here, we sought to examine the interrelationship between these previously established distance-related features of multisensory processing. Participants performed an audiovisual simultaneity judgment and redundant target task in near and far space, while audiovisual stimuli were presented at a range of temporal delays (i.e., stimulus onset asynchronies). In line with the previous findings, temporal acuity was poorer in near relative to far space. Furthermore, reaction time to asynchronously presented audiovisual targets suggested a temporal window for fast detection-a range of stimuli asynchronies that was also larger in near as compared to far space. However, the range of reaction times over which multisensory response enhancement was observed was limited to a restricted range of relatively small (i.e., 150 ms) asynchronies, and did not differ significantly between near and far space. Furthermore, for synchronous presentations, these distance-related (i.e., near vs. far) modulations in temporal acuity and multisensory gain correlated negatively at an individual subject level. Thus, the findings support the conclusion that multisensory temporal binding and gain are asymmetrically modulated as a function of distance from the observer, and specifies that this relationship is specific for temporally synchronous audiovisual stimulus presentations.

Keywords: Audiovisual; Depth; Gain; Multisensory Integration; Space; Temporal-binding window.

Fig. 1

Schematic bird’s-eye-view of the experimental setup. A, V, and AV stimuli were presented in the periphery or centrally in near (left) and far (right) space

Fig. 2

Performance on the SJ and MRT task as a function of SOA for near and far space. a Proportion of ‘simultaneous’ responses (i.e., reported synchrony) as a function of SOA and distance at which audiovisual stimuli were presented (black = near; red = far). b, c Group average for the PSS and TBW in near (black) and far (red) space. d Normalized multisensory gain as a function of SOA for near (black) and far (red) space. e, f Group average for the PFD and TWFD in near (black) and far (red) space. Significant differences are indicated with an asterisk (p < 0.05). Error bars represent standard error of the mean

Fig. 3

Average of median RTs in the AV condition (solid black line) in near (left panel) and far (right panel) space as a function of SOA between the auditory and visual stimuli. The average of median RTs in response to auditory (dark gray) and visual (light gray) targets are shown for comparison. Similarly, unisensory reaction times corrected for SOA (dashed shaded lines) are plotted for completeness. Finally, averaged median race model predictions are plotted (black dashed line) given the unisensory median reaction times corrected for SOA. Error bars indicate standard error of the mean

Fig. 4

Race model inequality violations (in ms) for responses to near (black) and far (red) audiovisual stimuli at SOAs of − 150 (leftmost) − 50 (second), 0 (third), and 50 (rightmost) ms. Significant violations are indicated with an asterisk (p < 0.05 corrected for nine tests using the Bonferroni method)

Fig. 5

Correlations between the change in TBW between near and far space and the number of deciles violating (> 0) the race model as a function of distance (Near–Far) for SOAs exhibiting a significant race model violation at the group level (leftmost: SOA = − 150 ms; second: SOA = − 50 ms; third: SOA = 0 ms; rightmost: 50 ms). None of these correlations were significant when considered in isolation (SOA = − 150 ms: r = − 0.01, p = 0.94; SOA = − 50 ms: r = − 0.06, p = 0.73; SOA = 0 ms: r = − 0.34, p = 0.056; and SOA = 50 ms: r = 0.04, p = 0.79), but the correlation at SOA = 0 ms is significant (rho = − 0.4, p = 0.02) when the co-variance at other SOAs is included in a partial correlation