Optimal multimodal integration in spatial localization

Abstract

Saccadic eye movements facilitate rapid and efficient exploration of visual scenes, but also pose serious challenges to establishing reliable spatial representations. This process presumably depends on extraretinal information about eye position, but it is still unclear whether afferent or efferent signals are implicated and how these signals are combined with the visual input. Using a novel gaze-contingent search paradigm with highly controlled retinal stimulation, we examined the performance of human observers in locating a previously fixated target after a variable number of saccades, a task that generates contrasting predictions for different updating mechanisms. We show that while localization accuracy is unaffected by saccades, localization precision deteriorates nonlinearly, revealing a statistically optimal combination of retinal and extraretinal signals. These results provide direct evidence for optimal multimodal integration in the updating of spatial representations and elucidate the contributions of corollary discharge signals and eye proprioception.

Figure 1.

An example of biologically plausible integration of efferent, afferent, and retinal signals. a, While looking at object A (red triangle, right), an observer plans a saccade toward object B (blue square), which was at the center of gaze at fixation 0, n saccades before (left). b, Likelihoods of independent estimates of object B's location in retinotopic coordinates. The afferent estimate, L̂_A, is proportional to the difference between the current eye position, e_N, and the position e₀ assumed by the eye during fixation on B. The efferent estimate, L̂_E, is proportional to the sum of all the saccades, s_k, which intervened between the two fixations. The retinal estimate, L̂_R, is determined by the position of object B on the retinal image. These three estimates can be combined to maximize the likelihood of localization. F_A and F_E represent mappings into retinotopic coordinates. This scheme is meant to provide an intuitive example of how independent estimates can be obtained and integrated in spatial localization. Several other plausible implementations of this strategy that do not necessary rely on retinotopic representations are conceivable.

Figure 2.

Experimental procedure and theoretical predictions. a, Two 20′ radius circles, the target and the response cue, were sequentially displayed at the center of gaze after n saccades (s₁ − s_n). Observers were asked to search for the two cues and report the remembered location of the target upon appearance of the response cue, either by placing a cursor (Experiment 1, visual localization) or by looking back (Experiment 2, oculomotor localization). The spatial positions of both the target and the response cue (X_T and X_R) varied across trials depending on the subject's eye movements. b–d, Predicted precision of different localization strategies. The variance of the localization error is expected to increase proportionally to the number of saccades between the target and the cue with a purely efferent mechanism of spatial updating (b), and to remain constant with a purely afferent one (c). d, e, The optimal integration method is to combine both sources, each weighted inversely to its variance (σ²_k). d, This strategy predicts that the localization error will first increase almost linearly and then saturate as saccades occur, since (e) weights are progressively reallocated from the corollary discharge (ω_E) to eye proprioception (ω_A). L̂_E, L̂_A, and L̂_O represent location estimates in a gaze-centered frame of reference.

Figure 3.

Characteristics of eye movements. Probability distributions of fixation durations (top) and saccade amplitudes (bottom) in the absence (left; data from Experiments 1 and 2 combined) and presence (center) of a visual reference. For comparison, fixation durations and saccade amplitudes measured during normal examination of a scene are also shown (right). In this condition, the same observers freely viewed pictures of natural scenes, each presented for 10 s. Data from all subjects were pooled together. The red line and number in each panel represent the mean of the distribution.

Figure 4.

Visual localization (Experiment 1). a, Summary of all trials. Each dot represents the localization error in an individual trial. Different panels show trials with different numbers of saccades between the target and the response cue. The mean error (red dot) and the 95% confidence ellipse are shown in each panel together with the marginal probability distributions and their best Gaussian fits, (μ, σ) (red curves). Data from all subjects (N = 4) were pooled together. b, Same data as in a after rotating the axes to align the abscissa with the cue-target direction. c, Mean dispersion area across subjects as a function of the number of saccades. Asterisks mark significant deviations (p < 0.001, two-tailed paired t tests), from the predictions of a purely efferent estimate, as given by the linear regression of the measurements obtained with the first three saccades (blue line). The black curve represents the least-squares fit of the ideal observer model. d, Optimal weighting of afferent and efferent estimates. As the number of saccades increases, proprioception is weighted more strongly and eventually becomes the predominant source of information. Error bars and shaded regions in c and d represent SEM.

Figure 5.

Individual subject data. The dispersion areas measured in the experiments are compared to the predictions of a corollary discharge model of spatial localization adjusted to take into account the finite size of the display and possible biases in the internal representation of saccades (blue line). The adjustment was obtained by means of Monte Carlo simulations of the individual experimental trials, in which the distribution of possible target positions, bounded by the monitor edges, was estimated by applying an individually fitted corollary discharge model to the recorded sequence of eye movements. Asterisks mark statistically significant differences between predicted and measured dispersion areas (p < 0.02, two-tailed t test). For comparison, the prediction of the corollary discharge model estimated on a random concatenation of saccades from different trials and without consideration of the monitor boundaries is also shown (dashed line). The least-squares parameters of the optimal integration model are shown in each panel (σ_E and σ_A). Error bars indicate SEM.

Figure 6.

Saccadic localization (Experiment 2). a, Mean dispersion area across subjects as a function of the number of saccades. Asterisks mark significant deviations (p < 0.05, two-tailed paired t tests) from the predictions of a purely efferent estimate, as given by the linear regression of the measurements obtained with the first three saccades (blue line). The black curve represents the least-squares fit of the ideal observer model. b, Optimal weighting of afferent and efferent estimates. Symbols and graphic conventions are the same as in Figure 4.

Figure 7.

Influence of a visual reference. a, Mean dispersion area across subjects as a function of the number of intervening saccades. The black curve represents the least-squares fit of the ideal observer model that integrates afferent, efferent, and retinal signals with the optimal weight combination (b). Conditions were identical to those of Experiment 1, except for the presence of a 5′ dot at the center of the display throughout each trial. Symbols and graphic conventions are the same as in Figure 4.

Figure 8.

Fixation duration. Comparison between the mean duration of the fixations in which the target was displayed (Target) and the mean duration of the other fixations in the search task (Others). Values represent averages across subjects. Data from Experiments 1 and 2 were combined. *p < 0.047 (two-tailed paired t test).