Meaning guides attention during scene viewing, even when it is irrelevant

Abstract

During real-world scene viewing, humans must prioritize scene regions for attention. What are the roles of low-level image salience and high-level semantic meaning in attentional prioritization? A previous study suggested that when salience and meaning are directly contrasted in scene memorization and preference tasks, attentional priority is assigned by meaning (Henderson & Hayes in Nature Human Behavior, 1, 743-747, 2017). Here we examined the role of meaning in attentional guidance using two tasks in which meaning was irrelevant and salience was relevant: a brightness rating task and a brightness search task. Meaning was represented by meaning maps that captured the spatial distribution of semantic features. Meaning was contrasted with image salience, represented by saliency maps. Critically, both maps were represented similarly, allowing us to directly compare how meaning and salience influenced the spatial distribution of attention, as measured by fixation density maps. Our findings suggest that even in tasks for which meaning is irrelevant and salience is relevant, meaningful scene regions are prioritized for attention over salient scene regions. These results support theories in which scene semantics play a dominant role in attentional guidance in scenes.

Keywords: Attention; Eye movements; Meaning; Salience; Scene perception.

Figure 1.. Trial structure for the two tasks.

The trial structure for (a) the brightness rating task and (b) the brightness search task.

Figure 2.. An example scene with the associated maps for each task.

(a) is an example scene with fixation locations from all participants in the rating task aggregated and overlaid. (b) is the fixation density map representing the example scene and fixation locations for the rating task. (c) is the example scene with fixations from the search task overlaid and (d) is the fixation density map representing the example scene and fixation locations in the search task. (e) is the center-biased meaning map and (f) is the unbiased meaning map for the example scene. (g) is the center-biased saliency map and (h) is the unbiased saliency map for the example scene.

Figure 3.. Scene-viewing times and number of fixations per trial for the brightness rating and brightness search tasks.

Distributions are shown for the scene viewing times of (a) the brightness rating and (b) brightness search tasks, and the number of fixations per trial of (c) the brightness rating and (d) brightness search tasks. Black dotted vertical lines represent the mean for each task.

Figure 4.. Response variability as a function of scene.

The average participant response and standard error of responses as a function of scene for (a) the brightness rating task and (b) the patch count task.

Figure 5.. Center biased maps.

Fixation density maps aggregated across subjects and scenes are shown for (a) the brightness rating task and (b) the brightness search task.

Figure 6.. Squared linear correlation and semi-partial correlation by scene for the brightness rating task.

Line plots show (a) the squared linear and (b) semi-partial correlations between the fixation density maps and meaning (red circles) and salience (blue squares) using center-biased meaning and saliency maps. Line plots also show (c) the squared linear and (d) semi-partial correlations using unbiased meaning and saliency maps. The scatter plots on the right show the grand mean (black horizontal line), 95% confidence intervals (colored boxes), and 1 standard deviation (black vertical line) for meaning and salience across all 40 scenes for each analysis.

Figure 7.. Squared linear correlation and semi-partial correlation by scene for the brightness search task.

The line plots show (a) the linear and (b) semi-partial correlations between fixation density and meaning (red circles) and salience (blue squares) for the search task using the center-biased meaning and saliency maps. Line plots also show (c) the linear and (d) semi-partial correlations for the search task using the unbiased meaning and saliency maps. The scatter plots on the right show the corresponding grand mean (black line), 95% confidence intervals (colored box), and one standard deviation (black vertical line) for meaning and salience across all scenes.

Figure 8.. Fixation by fixation time-step analyses for the brightness rating task.

The line plots show (a) the squared linear and (b) semi-partial correlations between fixation density and meaning (red circles) and salience (blue squares) as a function of fixation number collapsed across scenes for the rating task using the center-biased maps. Line plots also show (c) the squared linear and (d) semi-partial correlations between fixation density and meaning (red circles) and salience (blue squares) as a function of fixation order using the unbiased maps. Error bars represent the standard error of the mean.

Figure 9.. Fixation by fixation time-step analyses for the brightness search task.

The line plots show (a) the squared linear and (b) semi-partial correlations between fixation density and meaning (red circles) and salience (blue squares) as a function of fixation number collapsed across scenes for the search task using the center-biased maps. Line plots also show (c) the squared linear and (d) semi-partial correlations between fixation density and meaning (red circles) and salience (blue squares) as a function of fixation order using the unbiased maps. Error bars represent the standard error of the mean.

Figure 10.. Squared linear correlation and squared semi-partial correlation as a function of saccade amplitude to fixation.

The saccade amplitude results for the rating task are shown in the first column (a through e) in which (a) shows a histogram of saccade amplitude frequencies and average saccade amplitude (black dotted line), (b) and (d) show the squared linear and (c) and (e) show the semi-partial correlations between meaning (red circles) and saliency (blue squares) and fixation density as a function of saccade amplitude percentiles prior to fixation for the center-biased maps (b and d) and the unbiased maps (c and e). The second column (f through j) shows the saccade amplitude results for the search condition in which (d) shows a histogram of saccade amplitude frequencies and the average saccade amplitude (black dotted line), (g) and (i) show the squared linear correlations and (h) and (j) show the semi-partial correlations between meaning (red circles) and saliency (blue squares) with fixation density as a function of saccade amplitude percentiles using the center-biased maps (g and h) and the unbiased maps (i and j). Data points are averaged across all 40 scenes at each decile. Error bars represent standard error of the mean.