Don't hide the instruction manual: A dynamic trade-off between using internal and external templates during visual search

Abstract

Visual search is typically studied by requiring participants to memorize a template initially, for which they subsequently search in a crowded display. Search in daily life, however, often involves templates that remain accessible externally, and may therefore be (re)attended for just-in-time encoding or to refresh internal template representations. Here, we show that participants indeed use external templates during search when given the chance. This behavior was observed during both simple and complex search, scaled with task difficulty, and was associated with improved performance. Furthermore, we show that participants used external sampling not only to offload memory, but also as a means of verifying whether the template was remembered correctly at the end of trials. We conclude that the external world may not only provide the challenge (e.g., distractors), but may dynamically ease search. These results argue for extensions of state-of-the-art models of search, because external sampling seems to be used frequently, in at least two ways and is actually beneficial for task performance. Our findings support a model of visual working memory that emphasizes a resource-efficient trade-off between storing and (re)attending external information.

Figure 1.

(a) Sequence of a trial. Trials could contain either one template or four templates. In the unlimited access conditions, templates would remain on the screen throughout a trial. In the limited access conditions, templates would disappear as soon as search started. The vertical line was always present throughout a trial. Stimulus size is not to scale. (b) The eight Landolt C’s used in Experiment 1. (c) The eight original stimuli used in Experiment 2 (Arnoult, 1956). Each stimulus could be shown in one of four rotations, thus creating 32 stimuli. All stimuli occupied approximately 1.5° of visual angle.

Figure 2.

Outcome measures of Experiment 1 (left) and Experiment 2 (right). (a, d) The number of times the gaze crossed from the search area to the template area, as a measure of (re)sampling. Because each trial started with a central fixation cross, the minimum number of crossings was always 1, and any value above is indicative of resampling. (b, e) Balanced Accuracy, which takes into account an unequal proportion of target-present and -absent trials. Chance performance = 0.5. (c, f) Trial completion time in seconds, measured from trial start until keypress. Note: All panels except b and e visualize data of correctly answered and target-present trials only. Diamond markers denote individual participants.

Figure 3.

Increased dwell times and fixation durations when participants could not resample. (a, c) Time spent sampling the template area (in seconds; sum of fixation durations). (b, d) Median duration of fixations in the template area (in milliseconds) as a measure of the attempted depth of encoding of individual templates. Note: All panels visualize data of correctly answered and matching trials only. Diamond markers denote individual participants.

Figure 4.

Resampling could be used to refresh the template representation in VWM or to encode templates just-in-time. (a) The onset of crossings toward the template area, expressed as a percentage of trial duration. (b) The number of unique templates that were fixated per crossing. In the 1-Unlimited condition, only one unique template could be fixated. Note: Outcome measures of Experiments 1 and 2 combined. Diamond markers denote individual participants. Not all participants made multiple crossings in all trials.

Figure 5.

The ability to resample was used advantageously in both one- and four-template conditions, and in target-absent and target-present trials. (a) The number of gaze crossings per trial as an indicator of the degree of resampling. Split by condition and by target presence. (b) The percentage of trials in which the gaze ended in the template area, as an indicator of “double-checking” behavior. Split by condition and by incorrectly/correctly-answered trials. Note: Markers denote individual participants, aggregated over all trials. Bars display medians over participants with 95% CIs around the mean. † Results from a one-sample t test against 0.