Quantifying task-related gaze

Abstract

Competing theories attempt to explain what guides eye movements when exploring natural scenes: bottom-up image salience and top-down semantic salience. In one study, we apply language-based analyses to quantify the well-known observation that task influences gaze in natural scenes. Subjects viewed ten scenes as if they were performing one of two tasks. We found that the semantic similarity between the task and the labels of objects in the scenes captured the task-dependence of gaze (t(39) = 13.083; p < 0.001). In another study, we examined whether image salience or semantic salience better predicts gaze during a search task, and if viewing strategies are affected by searching for targets of high or low semantic relevance to the scene. Subjects searched 100 scenes for a high- or low-relevance object. We found that image salience becomes a worse predictor of gaze across successive fixations, while semantic salience remains a consistent predictor (X²(1, N=40) = 75.148, p < .001). Furthermore, we found that semantic salience decreased as object relevance decreased (t(39) = 2.304; p = .027). These results suggest that semantic salience is a useful predictor of gaze during task-related scene viewing, and that even in target-absent trials, gaze is modulated by the relevance of a search target to the scene in which it might be located.

Keywords: Eye movements: cognitive; Natural image statistics; Visual search.

Fig. 1

Example of a scene presented to a subject (A), the semantic salience heatmap for the task presented to the subject (Matched) (B), and the semantic salience heatmap for the task not presented to the subject (unmatched) (C). Red X’s represent subject’s gaze (note that in unmatched case (C), gaze is reproduced from matched case (B), as subject did not perform the unmatched task)

Fig. 2

Example of a presented scene (A) and its corresponding image salience (GBVS; B) and semantic salience (GloVe; C) heatmaps

Fig. 3

Average salience scores at fixation points for matched and unmatched cases. Gray lines represent difference in average scores for individual subjects. Black line represents mean decrease. Red lines represent median values

Fig. 4

As Fig. 3, for AUROC (area under the receiver operating characteristic curve) scores

Fig. 5

Average salience scores at fixation points for image salience (blue) and semantic salience (red), separated across target-absent objects with high semantic salience and target-absent objects with low semantic salience. Light blue and red lines represent trends within individual subjects. Red lines within boxplots represent median values. Black lines connecting box plots represent mean values

Fig. 6

As Fig. 5, for AUC (area under the curve) scores

Fig. 7

Salience scores for image salience (blue) and semantic salience (red) across number of fixations. For figure simplicity, only fixation numbers 1 through 10 are displayed. Boxplots represent summary statistics of all subjects at each fixation number. White circles with black centers represent median values. Black lines connecting boxplots represent mean values. Unfilled circles represent outliers