Faces in places: humans and machines make similar face detection errors

Abstract

The human visual system seems to be particularly efficient at detecting faces. This efficiency sometimes comes at the cost of wrongfully seeing faces in arbitrary patterns, including famous examples such as a rock configuration on Mars or a toast's roast patterns. In machine vision, face detection has made considerable progress and has become a standard feature of many digital cameras. The arguably most wide-spread algorithm for such applications ("Viola-Jones" algorithm) achieves high detection rates at high computational efficiency. To what extent do the patterns that the algorithm mistakenly classifies as faces also fool humans? We selected three kinds of stimuli from real-life, first-person perspective movies based on the algorithm's output: correct detections ("real faces"), false positives ("illusory faces") and correctly rejected locations ("non faces"). Observers were shown pairs of these for 20 ms and had to direct their gaze to the location of the face. We found that illusory faces were mistaken for faces more frequently than non faces. In addition, rotation of the real face yielded more errors, while rotation of the illusory face yielded fewer errors. Using colored stimuli increases overall performance, but does not change the pattern of results. When replacing the eye movement by a manual response, however, the preference for illusory faces over non faces disappeared. Taken together, our data show that humans make similar face-detection errors as the Viola-Jones algorithm, when directing their gaze to briefly presented stimuli. In particular, the relative spatial arrangement of oriented filters seems of relevance. This suggests that efficient face detection in humans is likely to be pre-attentive and based on rather simple features as those encoded in the early visual system.

Figure 1. Results of Experiment 1.

From left to right: percentage of trials in which illusory faces (IF) are selected over non faces (NF), real faces (RF) over illusory faces, and real faces over non faces in experiment 1. Black circles: individual observers (500 trials per observer in each condition); blue squares: mean over all 8 observers. IFs are selected over NFs above chance by all observers and significantly above chance on average; all observers select fewer RFs over IFs than over NFs, and significantly so on average.

Figure 2. Stimulus Ranking and Average.

A) Real face stimuli sorted by the probability of being correctly selected in experiment 1; middle row stimuli used in Experiment 2 B) illusory face stimuli sorted by the probability of being selected as face in experiment 1; middle row stimuli used in experiment 2 C) non face stimuli sorted by the probability of being selected as face in experiment 1. D) average of real face stimuli, E) average of illusory face stimuli, F) average of non face stimuli; averages in panel D–F are computed as pixelwise arithmetic means for each color channel, for an illustration how this average yields the face-like appearance in panel E (figure S1); G) real face stimulus with first two Haar filters of the OpenCV implementation of the Viola-Jones algorithm overlaid. The white squares indicate the area designated a face by the algorithm.

Figure 3. Results of Experiment 2.

Correct detection of real faces, when paired with illusory faces (% of trials). Both stimuli could be either upright or rotated (90°), resulting in a 2×2 design. The x-axis denotes the orientation of the real face, the marker the orientation of the illusory face. Black: experiment with grayscale stimuli, Blue: with colored stimuli. Error bars denote standard errors of the mean over 8 observers, who each performed 100 trials of each type. Real faces are detected better when upright than when rotated in any condition (all data are higher on the left than on the right); illusory faces are better distractors when presented upright (solid lines are always below dotted lines, i.e. real faces are detected worse, when the distractor is upright); performance is better (illusory faces are less frequently mistaken for real faces) for colored stimuli than for grayscale stimuli (blue datapoints consistently above black datapoints).

Figure 4. Results of Experiment 3.

From left to right: percentage of trials in which illusory faces (IF) are selected over non faces (NF), real faces (RF) over illusory faces, and real faces over non faces in experiment 3 (report with manual response). Notation and markers as in figure 1 - black circles: individual observers; blue squares: mean over all 8 observers. Illusory faces are neither preferred over non faces nor are illusory faces mistaken more frequently for real faces than are non faces. The effects observed in experiment 1 with eye-movement responses (figure 1) are thus absent for the manual response used here.