Modeling task control of eye movements

Abstract

In natural behavior, visual information is actively sampled from the environment by a sequence of gaze changes. The timing and choice of gaze targets, and the accompanying attentional shifts, are intimately linked with ongoing behavior. Nonetheless, modeling of the deployment of these fixations has been very difficult because they depend on characterizing the underlying task structure. Recently, advances in eye tracking during natural vision, together with the development of probabilistic modeling techniques, have provided insight into how the cognitive agenda might be included in the specification of fixations. These techniques take advantage of the decomposition of complex behaviors into modular components. A particular subset of these models casts the role of fixation as that of providing task-relevant information that is rewarding to the agent, with fixation being selected on the basis of expected reward and uncertainty about environmental state. We review this work here and describe how specific examples can reveal general principles in gaze control.

Figure 1. Overall cognitive system model

[1] While driving, multiple independent modules compete for gaze to improve their state estimates. [2] The winner, in this case Car Following, uses a fixation to improve its estimate of the followed car's location [3] The driver's action is selected according to the policy learned through RL. [4] The consequences of action selection are computed via the car simulator.

Figure 2. Evolution of state variable and uncertainty information for three modules

A Depiction of an update of the Follow module. Starting in the initial state, the necessary variables are known, but noise causes them to drift. According to the model reward uncertainty measurements, the Follow module is selected for a gaze update. This improves its state estimate while the other modules' state estimates drift further. B. A progression of state estimates for the three modules: constant speed maintenance, leader following and lane following. In each, the lines indicate state estimate vs. time for that module's relevant variable, in scaled units. Thus, for the speed module, the y-axis depicts the car's velocity, for the follow module it depicts the distance to a setpoint behind the lead car, and for the lane module it shows the angle to the closest lane center. If estimates overlap into a single line, the module has low uncertainty in its estimate. If estimates diverge, making a ‘cloud’, the module has high uncertainty. An update from the simulation for the Follow module can be seen at ten seconds. The fixation, indicated by pale shaded rectangle, lasts for 1.5 seconds. The figure shows how the individual state estimates drift between looks and how the state variables are updated during a look. The colored transparent region shows the noise estimate for each module. C. (top) Fixation on speedometer; (bottom) fixation on car.

Figure 3. Recognized Steps in peanut butter and jelly sandwich making

A computational model uses Bayesian evidence pooling to pinpoint steps in the task by observing the sandwich constructor's actions. The algorithm has access to the central one degree of visual input centered at the gaze point, which is delimited by the crosshairs. Also the position and orientation of each wrist is measured. The label in the upper left is the algorithm's estimate of the stage in the task.

Figure 4. Dynamic Bayes Network details

A. The dynamic Bayes network model of human gaze use factors the task variables and associated conditionally independent variables into a complex network. The network connectivity dictates how peripheral information can be propagated back to estimate the task variables. Although peripheral estimates can be very noisy, the task stages can be estimated very reliably. Filled circles denote measured data and open circles denote variables whose values must be computed. Bayes rule governs the propagation process. For example, the joint probability P(Gaze object, ClassifiedObject) can be factored as P(Gaze_object)P(ClassifiedObject| Gaze object). Bayes rule is then used to compute the probability of the gaze object given a measurement of the recognized object. Data recorded from previous sandwich makers is used to estimate the non-peripheral conditional probabilities. Successive time intervals are combined into a single ongoing task estimate, which results in the labels displayed in Fig. 3. B. The model classifies color information from the fixation point into one of the four classes of ClassifiedObject for each interval, but the results are very noisy C. The measurements for hand motion are classified into Hand Translating(eg reaching), Hand Rotating(eg screwing)for a fraction of the temporal intervals associated with stable gaze. Nonetheless, this information, when combined with the visual classifications and internal task priors, proves sufficient for task estimation.