Visual similarity effects in categorical search

Abstract

We asked how visual similarity relationships affect search guidance to categorically defined targets (no visual preview). Experiment 1 used a web-based task to collect visual similarity rankings between two target categories, teddy bears and butterflies, and random-category objects, from which we created search displays in Experiment 2 having either high-similarity distractors, low-similarity distractors, or "mixed" displays with high-, medium-, and low-similarity distractors. Analysis of target-absent trials revealed faster manual responses and fewer fixated distractors on low-similarity displays compared to high-similarity displays. On mixed displays, first fixations were more frequent on high-similarity distractors (bear = 49%; butterfly = 58%) than on low-similarity distractors (bear = 9%; butterfly = 12%). Experiment 3 used the same high/low/mixed conditions, but now these conditions were created using similarity estimates from a computer vision model that ranked objects in terms of color, texture, and shape similarity. The same patterns were found, suggesting that categorical search can indeed be guided by purely visual similarity. Experiment 4 compared cases where the model and human rankings differed and when they agreed. We found that similarity effects were best predicted by cases where the two sets of rankings agreed, suggesting that both human visual similarity rankings and the computer vision model captured features important for guiding search to categorical targets.

Figure 1.

Screen-shot of a teddy bear trial from Experiment 1.

Figure 2.

The number of objects corresponding to 60%, 70%, and 80% levels of inter-subejct agreement for each of the five visual similarity rankings (1 = least similar; 5 = most similar). (A) Teddy bears. (B) Butterflies.

Figure 3.

Representative target and non-target objects. (A) Teddy bears, (B) butterflies, (C) high similarity to teddy bears, (D) low similarity to teddy bears, (E) high similarity to butterflies, (F) low similarity to butterflies.

Figure 4.

Percentage of mixed condition trials in which the first object fixated was ranked as having a low, medium, or high target-distractor similarity for (A) Experiment 2 and (B) Experiment 3. Error bars show one standard error. Dashed lines indicate chance.

Figure 5.

Percentage of high-similarity (A) and low-similarity (B) trials in which the first object fixated was a human-only, model-only, human+model, or medium distractor in Experiment 4. Error bars show one standard error. Dashed lines indicate chance.