History Modulates Early Sensory Processing of Salient Distractors

Abstract

To find important objects, we must focus on our goals, ignore distractions, and take our changing environment into account. This is formalized in models of visual search whereby goal-driven, stimulus-driven, and history-driven factors are integrated into a priority map that guides attention. Stimulus history robustly influences where attention is allocated even when the physical stimulus is the same: when a salient distractor is repeated over time, it captures attention less effectively. A key open question is how we come to ignore salient distractors when they are repeated. Goal-driven accounts propose that we use an active, expectation-driven mechanism to attenuate the distractor signal (e.g., predictive coding), whereas stimulus-driven accounts propose that the distractor signal is attenuated because of passive changes to neural activity and inter-item competition (e.g., adaptation). To test these competing accounts, we measured item-specific fMRI responses in human visual cortex during a visual search task where trial history was manipulated (colors unpredictably switched or were repeated). Consistent with a stimulus-driven account of history-based distractor suppression, we found that repeated singleton distractors were suppressed starting in V1, and distractor suppression did not increase in later visual areas. In contrast, we observed signatures of goal-driven target enhancement that were absent in V1, increased across visual areas, and were not modulated by stimulus history. Our data suggest that stimulus history does not alter goal-driven expectations, but rather modulates canonically stimulus-driven sensory responses to contribute to a temporally integrated representation of priority.SIGNIFICANCE STATEMENT Visual search refers to our ability to find what we are looking for in a cluttered visual world (e.g., finding your keys). To perform visual search, we must integrate information about our goals (e.g., "find the red keychain"), the environment (e.g., salient items capture your attention), and changes to the environment (i.e., stimulus history). Although stimulus history impacts behavior, the neural mechanisms that mediate history-driven effects remain debated. Here, we leveraged fMRI and multivariate analysis techniques to measure history-driven changes to the neural representation of items during visual search. We found that stimulus history influenced the representation of a salient "pop-out" distractor starting in V1, suggesting that stimulus history operates via modulations of early sensory processing rather than goal-driven expectations.

Keywords: attentional selection; fMRI; priority; salience; visual search.

Figure 1.

Visual search task stimuli. On each trial, participants viewed a 4-item array and reported the orientation of the line inside the diamond-shaped target (horizontal or vertical). A, In the color constant condition, colors of targets and singleton distractors were fixed throughout the run. B, In the color variable condition, colors of targets and singleton distractors swapped randomly from trial to trial. C, An example trial with labels for the target, singleton distractor, and nontarget items.

Figure 2.

Behavioral capture during the visual search task. A, In the main MRI experiment (Experiment 1a), participants were significantly captured by the salient singleton distractor in the color variable condition, but not in the color constant condition. B, This pattern replicated in the behavior-only experiment (Experiment 1b). C, D, Capture costs (RT difference for distractor present – absent trials) were significantly larger in the color variable than in the color constant condition in Experiment 1a (C) and Experiment 1b (D). E, F, Capture costs (RT difference for distractor present – absent trials) were significantly modulated by the distance between the target and distractor in the color variable condition both in Experiment 1a (E) and Experiment 1b (F). Violin plot shading represents range and distribution of the data. Dots represent single subjects. Black error bars indicate ±1 SEM. Significance for uncorrected post hoc comparisons between adjacent bars within each experiment: *p < 0.05; **p < 0.01; ***p < 0.001.

Figure 3.

Univariate response in voxels selective to each item position, as a function of item type (target, singleton distractor, nontargets) and ROI. Violin plot shading represents range and distribution of the data. Dots represent single subjects. Black error bars indicate ±1 SEM.

Figure 4.

Single-item model estimates training and testing within the independent mapping task. A, Independent mapping task used to train the model to estimate spatial position of 4 search array items. Participants viewed a flickering checkerboard, which could appear at 1 of 24 positions around an imaginary circle. B, Blue lines indicate model estimates of viewed spatial position training and testing within the independent mapping task. Single-trial model estimates for each subject are aligned to 0 degrees and averaged. Black lines indicate model estimates for shuffled training labels. Opaque lines indicate group average. Semi-transparent lines indicate individual subjects. C, Descriptive statistics for best fit von Mises parameters (precision [κ], amplitude, baseline) to model estimates in B. Error bars indicate ±1 SEM. Opaque line indicates the group average. Semi-transparent lines indicate individual subjects.

Figure 5.

Linear classifier posterior probabilities. A, Posterior probabilities of each position bin chosen by a linear classifier (classify.m, diagLinear covariance option). B, Best-fitting von Mises parameters when fitting a von Mises to the posterior probabilities obtained by the linear classifier. A, B, Error bars indicate ±1 SEM. Opaque line indicates the group average. Semi-transparent lines indicate individual subjects. These parameters closely parallel changes to IEM estimates across ROIs, but with a change in scale, as shown in C. C, Correlation between the κ parameter for a von Mises fit to IEM output versus linear classifier posterior probabilities.

Figure 6.

Generating predictions for 4-item model estimates by averaging single-item model estimates from the independent mapping task. A, Average from the independent mapping task plotted at 4 hypothetical item locations. Here, these 4 “items” are represented with equal priority. B, Hypothetical observed response when measuring a single trial containing the 4 items presented simultaneously. This line is the average of all lines in A. C, Same as in A and B, but with the item at position 0 assigned a higher response amplitude than the other three items. D, Same as in A and B, but with both an enhanced item at position 0 and a suppressed item at position −90. E, Actual IEM model output for 4-item search arrays in V2 (target plotted at 0, distractor plotted at −90). To estimate the strength of each of the 4 underlying item representations, one can simply measure the height (a.u.) at expected item peaks (i.e., −180, −90, 0, and 90). Alternatively, one may use a non-negative least-squares solution to estimate weights for a regressor for each of the 4 item positions. Each regressor is the 1-item IEM output from the independent mapping task within the same region (e.g., V2), shifted to the appropriate item location. F, Example IEM output and best-fitting non-negative least-squares solution with 4 item regressors.

Figure 7.

Dissociable effects of stimulus history on target enhancement and distractor suppression. A, Model responses for individual ROIs as a function of task condition (arrays with target-distractor distance ±90). Purple and green lines indicate the output of the IEM in the color constant and color variable conditions, respectively. Shaded error bars = 1 SEM. Background panels at −180°, −90°, 0°, and 90° show the positions of the 4 search array items: blue represents target (T); pink represents distractor (D); gray represents nontarget items (N1 and N2). Target enhancement can be seen as the greater height at position 0: The IEM peak at the blue bar is higher than the IEM peak at the other bars. History-driven distractor suppression can be seen as the lower height at position −90 for the color constant versus color variable conditions: The IEM peak at the pink bar is higher for the green line than for the purple line. B, Target amplitude as a function of ROI and task condition. There was no effect of task condition on target amplitude, but a significant increase in target amplitude across ROIs. Violin plot shading represents range and distribution of the data. Dots represent single subjects. Black error bars indicate ±1 SEM. C, Distractor amplitude as a function of ROI and task condition. There was a significant effect of task condition on distractor amplitude, and this history-driven effect did not interact with ROI.

Figure 8.

No effect of history on arrays with insufficient target-distractor competition. Model estimates for all channels for arrays with target-distractor distance of ±180 degrees. Purple and green lines indicate the output of the IEM in the color constant and color variable conditions, respectively. Shaded error bars indicate 1 SEM. Background panels at −180°, −90°, 0°, and 90° show the positions of the 4 search array items: blue represents target (T); pink represents distractor (D); gray represents nontarget items (N1 and N2).

Figure 9.

Each target-present task condition versus a target absent baseline. A, B, Comparison of the color constant condition (A) and color variable condition (B) to a target absent baseline when the target-distractor distance was 90 degrees. C, D, Comparison of the color constant condition (C) and color variable condition (D) with a target absent baseline when the target-distractor distance was 180 degrees. Purple, green, and gray lines indicate the output of the IEM in the color constant, color variable, and distractor absent conditions, respectively. Shaded error bars indicate SEM. Background panels at −180°, −90°, 0°, and 90° show the positions of the 4 search array items: blue represents target (T); pink represents distractor (D); gray represents nontarget items (N1 and N2).

Figure 10.

Aggregate ROI analysis shows target enhancement and distractor suppression in both early visual and parietal cortex. A, B, Model estimates for all channels in early visual cortex (A) and parietal cortex (B) for arrays with target-distractor distance of ±90°. C, D, Model estimates for all channels in early visual cortex (C) and parietal cortex (D) for arrays with target-distractor distance of ±180°. Purple and green lines indicate the output of the IEM in the color constant and color variable conditions, respectively. Shaded error bars indicate 1 SEM. Background panels at −180°, −90°, 0°, and 90° show the positions of the 4 search array items: blue represents target (T); pink represents distractor (D); gray represents nontarget items (N1 and N2).

Figure 11.

Comparison of each distractor-present task condition to a distractor-absent baseline. A–D, Model estimates for all channels in early visual cortex (V1-V3) (A,B) and parietal cortex (IPS0-IPS3) (C,D), comparing the color constant condition (distractor present, target-distractor distance ±90) to a target absent baseline. E–H, Plots from A–D for the target-distractor distance ± 180 trials.

Figure 12.

Simplified diagram illustration of local-image versus temporal-integration salience for a simple image with one feature and location. A, In 2D salience computations, stimulus-driven stimulus drive is determined locally within a given image without respect to prior images. Sequence 1 is 4 different trials; and on each trial, the same stimulus is shown (blue-blue-blue-blue). Sequence 2 is 4 different trials, but the final trial is a different color from the preceding trials (green-green-green-blue). The final trial (blue) is physically identical for the two sequences. So, the final stimuli (trial n in each sequence) have identical 2D salience. Assuming that we chose equiluminant green and blue values, then each “frame” in the sequence likewise has approximately the same image-computable salience, as shown by the uniform-sized square pulses in the diagram. Alternatively, stimulus-driven salience maps may better be conceived of as reflecting a temporally integrated 3-D salience map, as early sensory neurons adapt to ongoing stimulus features. In Sequence 1 (blue-blue-blue-blue), the activity of neurons that are maximally responsive to blue wanes because of adaptation. In Sequence 2, the activity of neurons maximally responsive to green wanes over the first 3 trials, but the final stimulus elicits a robust response from the nonadapted blue-preferring neurons. Thus, temporally integrated salience for the trial n in each sequence differs across the two sequences, although the stimuli are physically identical. B, Most studies of predictive coding and adaptation consider changes to neural activity for a single item. Here, we illustrate how adaptation can have consequences for stimulus-driven saliency that arises from inter-item competition within multi-item arrays (e.g., Itti and Koch, 2000). Top, For the first presentation of the array, all neurons respond strongly, leading to classic inter-item competition effects that yield high distractor saliency. Bottom, With repeated presentations and adaptation, overall activity and inter-item competition are weakened, yielding a relative attenuation of the distractor.