Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Nov 4;18(1):nsad059.
doi: 10.1093/scan/nsad059.

Distinct patterns of neural response to faces from different races in humans and deep networks

Ao Wang  1   2 , Magdalena W Sliwinska  1   3 , David M Watson  1 , Sam Smith  1 , Timothy J Andrews  1
Affiliations

Distinct patterns of neural response to faces from different races in humans and deep networks

Ao Wang et al. Soc Cogn Affect Neurosci. .

Abstract

Social categories such as the race or ethnicity of an individual are typically conveyed by the visual appearance of the face. The aim of this study was to explore how these differences in facial appearance are represented in human and artificial neural networks. First, we compared the similarity of faces from different races using a neural network trained to discriminate identity. We found that the differences between races were most evident in the fully connected layers of the network. Although these layers were also able to predict behavioural judgements of face identity from human participants, performance was biased toward White faces. Next, we measured the neural response in face-selective regions of the human brain to faces from different races in Asian and White participants. We found distinct patterns of response to faces from different races in face-selective regions. We also found that the spatial pattern of response was more consistent across participants for own-race compared to other-race faces. Together, these findings show that faces from different races elicit different patterns of response in human and artificial neural networks. These differences may underlie the ability to make categorical judgements and explain the behavioural advantage for the recognition of own-race faces.

Keywords: DCNN; ORE; face; race.

PubMed Disclaimer

Conflict of interest statement

The authors declared that they had no conflict of interest with respect to their authorship or the publication of this article.

Figures

Fig. 1.
Fig. 1.
Examples of images from the different stimulus conditions.
Fig. 2.
Fig. 2.
Location of the face-selective regions following a group analysis across all participants. Regions of interest are superimposed on the MNI152 brain (x = 40, y = -−60, z = −16). FFA: fusiform face area, OFA: occipital face area, STS: posterior superior temporal sulcus, AMG: amygdala.
Fig. 3.
Fig. 3.
Similarity matrices from the images in the face matching task calculated from the 13 convolutional and 3 fully connected layers of VGG-Face. The similarity matrix shows the similarity (correlation) of all combinations of the 540 images in the stimulus set. The 540 images comprised 180 Asian, 180 Black and 180 White faces. The similarity of each image pair was calculated by correlating the DCNN feature vectors for pairs of images.
Fig. 4.
Fig. 4.
Sensitivity to decoding face races across different layers of VGG-Face. Filled symbols indicate decoding significantly higher than chance. Sensitivity to face race becomes most evident in the fully connected layers (Fc). However, there is also a greater sensitivity to White faces compared to Black and Asian faces in the later convolutional layers (Conv) of the DCNN.
Fig. 5.
Fig. 5.
Decoding of facial identity in VGG-Face. Plot illustrates the sensitivity to decoding same-identity versus different-identity face pairs for each layer in each of the different races. Filled symbols indicate significantly higher sensitivity to same-identity than different-identity face pairs. The ability to discriminate identity was greatest in the fully connected layers (Fc) and there was a greater sensitivity to White faces compared to Asian and Black faces.
Fig. 6.
Fig. 6.
(A) The correlation between pairwise image similarity in the DCNN and proportion of same identity judgements of Asian participants and (B) White participants was calculated for different layers in the DCNN. The dashed line indicates the critical r-value at P < 0.05. Significant correlations were most evident in the fully connected layers (14–16).
Fig. 7.
Fig. 7.
MVPA showing different spatial patterns of response to faces from different races in face regions. (A) The similarity in the spatial patterns of response between faces from the Same Race was compared to the similarity in the spatial patterns between faces from Different Races. (B) This shows a main effect of Face Race in all regions with more similar patterns of response to faces from the Same Race compared to Different Race (*** P < 0.001, * P < 0.05). Error bars represent standard error of the mean.
Fig. 8.
Fig. 8.
MVPA showing more similar spatial patterns of response to own-race compared to other-race faces in the OFA and FFA. (A) The spatial pattern of response between White or between Asian faces was compared in Asian and White participants. (B) There was an interaction between Face and Participant race in the OFA and FFA (*P < 0.05, n.s. not significant). This reflects the spatial pattern of response to own-race faces being more similar than the pattern of response to other-race faces in these regions. Error bars represent standard error of the mean.
Fig. 9.
Fig. 9.
fMR adaptation to faces from different races. There were no significant interactions between Face*Participant in any of the face-selective regions. This shows that the magnitude of adaptation was not modified by the race of the participants. Error bars represent SEM.
Fig. 10.
Fig. 10.
MVPA showing different spatial patterns of response to faces and pareidolic objects (A) The spatial pattern of response between different race faces (Face–Face) was compared to the spatial pattern between faces and pareidolic objects (Face–Object) in Asian and White participants. (B) The results reveal a significant effect of Category due to more similar patterns of response between faces (Face–Face) compared to the patterns between faces and objects (Face–Object). Error bars represent standard error of the mean. *** P < 0.001, ** P < 0.005.
Fig. 11.
Fig. 11.
Similar neural responses to pareidolic objects in Asian and White participants. Adaptation to pareidolic objects was evident in the OFA and FFA. However, there was no difference in the magnitude of adaptation to pareidolic objects between White and Asian participants. Error bars represent standard error of the mean.
See this image and copyright information in PMC

References

    1. Andrews T.J., Davies-Thompson J., Kingstone A., Young A.W. (2010). Internal and external features of the face are represented holistically in face-selective regions of visual cortex. Journal of Neuroscience, 30(9), 3544–52. - PMC - PubMed
    1. Andrews T.J., Ewbank M.P. (2004). Distinct representations for facial identity and changeable aspects of faces in the human temporal lobe. NeuroImage, 23(3), 905–13. - PubMed
    1. Bagnis A., Celeghin A., Diano M., et al. (2020). Functional neuroanatomy of racial categorization from visual perception: a meta-analytic study. Neuroimage, 217, 116939. - PubMed
    1. Brosch T., Bar-David E., Phelps E.A. (2013). Implicit race bias decreases the similarity of neural representations of black and white faces. Psychological Science, 24(2), 160–6. - PMC - PubMed
    1. Burton A.M., Bruce V., Dench N. (1993). What’s the difference between men and women? Evidence from facial measurement. Perception, 22(2), 153–76. - PubMed

LinkOut - more resources