Varying sex and identity of faces affects face categorization differently in humans and computational models

Abstract

Our faces display socially important sex and identity information. How perceptually independent are these facial characteristics? Here, we used a sex categorization task to investigate how changing faces in terms of either their sex or identity affects sex categorization of those faces, whether these manipulations affect sex categorization similarly when the original faces were personally familiar or unknown, and, whether computational models trained for sex classification respond similarly to human observers. Our results show that varying faces along either sex or identity dimension affects their sex categorization. When the sex was swapped (e.g., female faces became male looking, Experiment 1), sex categorization performance was different from that with the original unchanged faces, and significantly more so for people who were familiar with the original faces than those who were not. When the identity of the faces was manipulated by caricaturing or anti-caricaturing them (these manipulations either augment or diminish idiosyncratic facial information, Experiment 2), sex categorization performance to caricatured, original, and anti-caricatured faces increased in that order, independently of face familiarity. Moreover, our face manipulations showed different effects upon computational models trained for sex classification and elicited different patterns of responses in humans and computational models. These results not only support the notion that the sex and identity of faces are processed integratively by human observers but also demonstrate that computational models of face categorization may not capture key characteristics of human face categorization.

Figure 1

Examples of face stimuli used in Experiment 1. Left column, original faces of a female (top) and a male (bottom). Right column, their modified versions, showing the faces in opposite-sex appearance. Written informed consent was obtained from our colleagues to publish images of their original and manipulated faces in an online open access publication.

Figure 2

Mean accuracy data in Experiment 1 for each face set. Error bars represent standard errors of the means (SEMs). Long horizontal bars indicate significance of interaction, and short bars indicate significance of follow-up contrast. *p < 0.01; ^†p < 0.10.

Figure 3

Model-based sex categorization performance in Experiment 1. The results of the control group are replotted here to illustrate the pattern of response observed with humans. Long horizontal bars indicate significance of face manipulation effect, and short bars indicate significance of follow-up contrast. **p < 0.001; *p < 0.05. Error bars represent SEMs. Note that the error bars in some conditions (like those with near perfect performance) were too small to appear clearly here.

Figure 4

Examples of face stimuli used in Experiment 2. From left to right, anti-caricature (60% identity strength), original (100% identity strength), and caricature (140% identity strength) of a male face. Informed consent was obtained from our colleague to publish images of his original and manipulated faces in an online open access publication.

Figure 5

Mean accuracy data in Experiment 2 shown for each face set. Error bars represent SEMs. Horizontal bars indicate significance of difference between face manipulation conditions. **p < 0.005; *p < 0.05.

Figure 6

Sex classification performance of the computational models in Experiment 2. The results of the control group are replotted here to illustrate the pattern of response observed with humans. Horizontal bars indicate significance of difference following a main effect of face manipulation. **p < 0.01; *p < 0.05; ^†p < 0.10. Error bars represent SEMs. Note that the error bars in some conditions (like those with near perfect performance) were too small to appear clearly here.