Near optimal combination of sensory and motor uncertainty in time during a naturalistic perception-action task

Abstract

Most behavioral tasks have time constraints for successful completion, such as catching a ball in flight. Many of these tasks require trading off the time allocated to perception and action, especially when only one of the two is possible at any time. In general, the longer we perceive, the smaller the uncertainty in perceptual estimates. However, a longer perception phase leaves less time for action, which results in less precise movements. Here we examine subjects catching a virtual ball. Critically, as soon as subjects began to move, the ball became invisible. We study how subjects trade-off sensory and movement uncertainty by deciding when to initiate their actions. We formulate this task in a probabilistic framework and show that subjects' decisions when to start moving are statistically near optimal given their individual sensory and motor uncertainties. Moreover, we accurately predict individual subject's task performance. Thus we show that subjects in a natural task are quantitatively aware of how sensory and motor variability depend on time and act so as to minimize overall task variability.

FIG. 1.

Overview of the ball catching experiment. A: the ball is launched from a central position with random horizontal velocity and is accelerated by artificial gravity toward the ground (green line); the total task time to touchdown is therefore constant. Subject controls a veridically aligned paddle of varying width w (the subjects hand center underneath the flat screen is matched to the paddle center). After the ball is launched, the subject chooses when to start moving the paddle from the far right start position (x = 0). Once paddle movement begins, the ball becomes invisible as it falls toward the ground. At all times, a thin horizontal line indicates the vertical position of the ball. A successful catch occurs when the ball center touches ground level within the width of the paddle; otherwise, to provide feedback, the ball is rendered visible and continues its trajectory until it disappears at the bottom of the screen. B: time and uncertainty relationships in the ball catching experiment. In general, sensory uncertainty (green curve and axis) decreases with time, and so does motor uncertainty (blue curve and axis). However, because of the design of the experiment, the more task time is spent sensing, the less task time remains for movement. Therefore as task time progresses and the ball approaches ground level, motor uncertainty increases. Thus the switching between sensory and motor uncertainty causes a combined uncertainty (black curve) with a globally minimal task error at the accordingly optimal switching time. C: task error (measured as ball landing position minus final paddle position) SD plotted vs. trial [batch of 20 trials for each subject (blue crosses), mean across 11 subjects (blue line), and SE (shaded region)]. The horizontal level line represents the minimal task error across subjects if they would have used the optimal switching times for each trial.

FIG. 2.

Construction of the task variability surface from the sensory estimation and motor variability experiments. A: the time dependency of the sensory error (means ± SE) is fitted by square root power-law for each of the 11 subjects. B: the time- and position-dependent motor error is fitted by Fitt's law; here the time-dependency fit (curves) is shown alone by averaging over all landing positions. A and B show the data (means ± SE) for each of the 11 subjects. The sensory and motor error SD is converted into variances by squaring. C: the sensory error variance surface is constructed by interpolating the fitted sensory error relationship for all possible switching times (the switching time corresponds to the time available for sensing) and landing positions. D: the motor error surface is obtained as in C by using Fitt's law to interpolate the motor variability SD and squaring the values. Note that in the ball catching experiment, we equate the movement time of the motor experiment with the remaining task time after switching. We plot the motor error in terms of the switching time t and therefore the motor error increases in the direction of the switching time axis. E: we assume that sensor and motor error (surfaces) are independent of each other and we can thus obtain an estimate of the ball catching task variability surface by summing both C and D together. This combined error surface resembles a “river valley” with a minimum task error curve (white line) that shows the optimum switching time for every ball landing position.

FIG. 3.

Switching times and task variability surfaces for all 11 subjects. A–K: each subplot shows the behavior for each subject and the combined variance. Individual trials of the ball catching experiment are shown as white dots, marking the distance of the paddle's initial position to the ball's landing position (i.e., the required movement distance) and the chosen switching time. The same color coding scale (log units of variance) is used for all subjects to highlight the individuality of each subject's error surface. The task variability surfaces (see text and Fig. 2) are computed from the sum of the sensor and motor error surfaces, as determined independently for each subject in the sensory and motor estimation experiment. The optimal switching time curve (white curve) lies in the valley (blue region) of the minimum of the task variability surface. Subjects in A–C, H, and I are naïve members of the laboratory.

FIG. 4.

Switching times and task variability surfaces for 6 different subjects in the faster and more difficult sequence-controlled version of the experiments (A–F). Individual trials of the ball catching experiment are shown as white dots, marking the distance of the paddle's initial position to the ball's landing position (i.e., the required movement distance) and the chosen switching time together with the optimal switching time curve (white line). The same color coding scale (log units of variance) and conventions apply as in Fig. 3. Note how the overall level of the combined error surface is higher in this version of the task than that one in Fig. 3, reflecting increased difficulty as subjects have to act within higher sensory and motor uncertainties. Note subject A had difficulty in completing fast movements (<500 ms) over the full range of possible landing positions. All subjects were naïve.

FIG. 5.

Catch probability given paddle width (data pooled over all landing positions) for all 11 subjects (arranged to correspond with Fig. 3). A–K: each plot shows the optimal catch performance achieved by using optimal switching times (black curve) and subject's actual catch performance (circles) for each paddle width.