Neural computations in prosopagnosia

Abstract

We report an investigation of the neural processes involved in the processing of faces and objects of brain-lesioned patient PS, a well-documented case of pure acquired prosopagnosia. We gathered a substantial dataset of high-density electrophysiological recordings from both PS and neurotypicals. Using representational similarity analysis, we produced time-resolved brain representations in a format that facilitates direct comparisons across time points, different individuals, and computational models. To understand how the lesions in PS's ventral stream affect the temporal evolution of her brain representations, we computed the temporal generalization of her brain representations. We uncovered that PS's early brain representations exhibit an unusual similarity to later representations, implying an excessive generalization of early visual patterns. To reveal the underlying computational deficits, we correlated PS' brain representations with those of deep neural networks (DNN). We found that the computations underlying PS' brain activity bore a closer resemblance to early layers of a visual DNN than those of controls. However, the brain representations in neurotypicals became more akin to those of the later layers of the model compared to PS. We confirmed PS's deficits in high-level brain representations by demonstrating that her brain representations exhibited less similarity with those of a DNN of semantics.

Keywords: EEG; RSA; artificial neural networks; prosopagnosia; semantic representations.

Fig. 1

Overview of the experiment. a) the histogram shows the Cambridge face memory test long-form (CFMT+, (Russell et al. 2009) scores for PS, our typical recognizers (dark gray bars), and an additional 332 neurotypical observers from three independent studies for comparison (Faghel-Soubeyrand et al. 2019; Tardif et al. 2019; Fysh et al. 2020). b) Participants were engaged in a one-back task while their brain activity was recorded with high-density electroencephalography. The stimuli included objects from various categories, such as faces, objects, and scenes. Note that the face drawings shown here are anonymized representations used as substitutes for the actual face stimuli presented to our participants. c) Representational similarity analyses consisted in constructing brain representational dissimilarity matrices (RDMs) by comparing representational patterns (as characterized by EEG topographies) for all pairwise comparisons of stimuli, independently for each time-point and participants. Specifically, RDMs were constructed using cross-validated decoding performance between the EEG topographies at 4 ms intervals, providing a dynamic account of representational geometries unfolding after stimulus onset. d) To evaluate the temporal evolution of brain representations, a temporal generalization matrix was computed for each participant. This involved calculating all pairwise correlations between a participant’s time-resolved brain RDMs. A specific time-resolved brain RDM is considered to “generalize” to later time-resolved brain RDMs when it exhibits a positive correlation with them.

Fig. 2

Temporal generalization of EEG representations across time in PS and controls. a) Temporal generalization over all pairwise stimulus comparisons. To assess the temporal evolution of brain representations, we computed a temporal generalization matrix for each participant. This process involved calculating pairwise correlations between time-resolved brain representational dissimilarity matrices (RDMs). The leftmost column displays the mean temporal generalization matrix of control participants. The yellowish square in the upper right section of the matrix indicates temporal generalization within the N170 time window. The central column illustrates the temporal generalization matrix of PS, which resembles that of the controls but is associated with earlier brain RDMs. This is most evident in the rightmost column, representing the difference between PS’s and controls’ temporal generalization matrices. Statistically significant regions in this contrast matrix are outlined in black (P < 0.05, uncorrected), with only positive differences reaching the threshold. b) Similar to a), temporal generalization matrices were computed, but this time specifically for a subset of time-resolved brain RDMs comparing pairs of face stimuli. c) Similar to a), temporal generalization matrices were computed, but this time specifically for a subset of time-resolved brain RDMs comparing pairs of face and nonface stimuli. d) Similar to a), temporal generalization matrices were computed, but this time specifically for a subset of time-resolved brain RDMs comparing pairs of nonface stimuli.

Fig. 3

Comparison of brain representations with those of artificial neural networks of visual and semantic processing. a) Partial spearman correlation between brain RDMs and ecoset-trained AlexNet RDMs (removing shared correlation between brain and semantic model) is shown for PS (pink curve) and controls (gray curve). Each column shows different layer RDMs in ascending order from left to right. See Supplementary Fig. 2 for the same analysis on all the deep neural networks (DNN) layers. We found lower similarity of visual computations within the brain of PS compared to controls in layers 6 and 7 (black dots indicate significant contrasts in favor of controls, Howell-Crawford modified t-tests, P < 0.05; uncorrected), with differences peaking in higher-level DNN layer 7. We observed the opposite in layers 1 to 3 and, to a lesser extent in layer 5, at early time points (red dots indicate significant contrasts in favor of PS, Howell-Crawford modified t-tests, P < 0.05; uncorrected). Similar results were observed when comparing brains and DNN models without removing the shared information between brains and the semantic (caption-level) model (Supplementary Fig. 4). See Supplementary Fig. 3a and b for partial Spearman correlations for AlexNet trained on ImageNet and of VGGFace, respectively. b) Partial Spearman correlation with RDMs of the semantic model (excluding shared information between brain and AlexNet) was significantly lower in the brain of PS compared to controls (cyan curve; black dots indicate significant contrasts, P < 0.05; uncorrected). Similar results were observed when comparing brains and DNN models without removing the shared information between brains and the semantic (caption-level) model (see Supplementary Fig. 5). The shaded areas of all curves represent the standard error for the controls.