The Neural Dynamics of Face Ensemble and Central Face Processing

Abstract

Extensive work has investigated the neural processing of single faces, including the role of shape and surface properties. However, much less is known about the neural basis of face ensemble perception (e.g., simultaneously viewing several faces in a crowd). Importantly, the contribution of shape and surface properties have not been elucidated in face ensemble processing. Furthermore, how single central faces are processed within the context of an ensemble remains unclear. Here, we probe the neural dynamics of ensemble representation using pattern analyses as applied to electrophysiology data in healthy adults (seven males, nine females). Our investigation relies on a unique set of stimuli, depicting different facial identities, which vary parametrically and independently along their shape and surface properties. These stimuli were organized into ensemble displays consisting of six surround faces arranged in a circle around one central face. Overall, our results indicate that both shape and surface properties play a significant role in face ensemble encoding, with the latter demonstrating a more pronounced contribution. Importantly, we find that the neural processing of the center face precedes that of the surround faces in an ensemble. Further, the temporal profile of center face decoding is similar to that of single faces, while those of single faces and face ensembles diverge extensively from each other. Thus, our work capitalizes on a new center-surround paradigm to elucidate the neural dynamics of ensemble processing and the information that underpins it. Critically, our results serve to bridge the study of single and ensemble face perception.

Keywords: EEG; face ensembles; neural dynamics; pattern analysis; shape cues; surface cues.

Figure 1.

Disentangling face shape and surface information. A, Sixty faces were marked with (B) 82 fiducial points accounting for facial shape. C, The coordinates of these points were extracted and (D) averaged as the mean shape. E, All faces were subsequently remapped to this mean shape, resulting in faces that varied only in surface properties. F, A pixelwise average of these faces yielded an average face. G, The average face could then be mapped to the fiducial points of each of the 60 faces to yield faces that vary in shape but not surface.

Figure 2.

Generating a stimulus matrix of faces that vary parametrically in shape and surface properties. A, PCA was separately applied to shape coordinates and to surface images previously derived from a set of faces (Fig. 1). B, Four cardinal anchors were generated as the endpoints of two dimensions, equidistantly from the average face in image space. C, The anchors were linearly interpolated (color-coded bars indicate the proportional contribution of each dimension and polarity to any given cell in the matrix). D, A matrix of stimuli varying in surface properties based on the stimulus design schema from (C). In the bottom formula, A are anchors, C are coefficients based on the x and y position of a face in the matrix, and μ is the average face. Four stimulus sets were created by extracting faces along the edges of the matrix (purple boxes). Only faces bounded by these boxes were used as experimental stimuli. Row and column averages are displayed next to the face matrix. This procedure was similarly applied to shape coordinates to yield stimuli that only varied in shape, which were subsequently combined with surface information to produce stimuli varying in both shape and surface properties.

Figure 3.

All single-face stimuli. There were four sets of faces, split across dimensions D1 and D2, and across plus and minus polarities. Six faces varied parametrically in shape, surface, or both. The mean is shown in the rightmost column of each panel (note that Faces 1 and 6 are shared between neighboring stimulus sets as a result of how they were extracted from the face matrix; Fig. 2D).

Figure 4.

Examples of ensemble stimuli from set D2 that vary in both shape and surface properties. Six faces form the ensemble surround, and the mean of the set of the same or opposite polarity provides the ensemble's central face. Consistent ensembles have matching centers and surrounds (e.g., D2+ surround, and the mean of D2+ as center). Inconsistent ensembles have opposing centers and surrounds (e.g., D2+ surround, and the mean of D2− as center).

Figure 5.

ERPs averaged across 12 OT channels for consistent ensemble, inconsistent ensemble, and single-face stimuli. Univariate differences were found between single faces and ensembles, but not between different types of ensembles. *p < 0.05, ***p < 0.001; Bonferroni-corrected.

Figure 6.

Temporally cumulative pairwise decoding of (A) single faces, (B) face ensembles, and (C) consistent versus inconsistent face ensembles. Significant decoding was observed in all cases except for decoding of only shape or only surface for consistency. Pairwise differences were significant for comparisons of shape versus surface (A,B), and there were no differences between shape and surface versus just surface. ns, not significant, *p < 0.05, **p < 0.01, ***p < 0.001; Bonferroni-corrected; error bars show ± 1 SE.

Figure 7.

The time course of pairwise decoding across attribute groups for (A) single faces and (B) face ensembles. Consistent with the temporally cumulative results (Fig. 6), surface information, by itself or combined with shape, contributed more extensively to decoding compared with shape by itself. Each position on the time axis represents accuracy at that given time point centered on a 10 ms window. Accuracy across time bins is compared against 50% (hashed line) via sign test. Resulting p values are FDR-corrected (q < 0.05), and significant time bins are denoted by dark gray shading; light-shaded gray indicates ± 1 SE

Figure 8.

Separate decoding of surround and center faces. A, Surround and center faces are decoded significantly above chance (collapsed across all three attribute conditions). B, Surround decoding is maximized for surface and shape properties combined. C, Central face decoding is driven by both shape and surface properties, with a higher contribution from surface information. D, The time course of decoding surround faces, center faces, and center-surround consistency is shown collapsed across attribute conditions. Both the ensemble surround and center, as well as their consistency, show intervals of above-chance classification. ns, not significant, *p < 0.05, **p < 0.01, ***p < 0.001; Bonferroni-corrected; error bars show ± 1 SE.

Figure 9.

A comparison of the temporal profiles for ensemble and single-face processing (relying on information from Figs. 7, 8D). Colored bands indicate significant temporal intervals of decoding. Brightness relates to decoding accuracy (brighter colors indicate higher accuracy) at a given time point, standardized to aid visual comparison. The temporal profile of surface information is similar for both single and ensemble faces, whereas that of shape vastly differs, with an earlier decoding onset and longer duration for ensembles compared with single faces. The profiles for single and ensemble center faces are also highly similar. Relating center and surround information (i.e., consistency) occurs across the entire epoch.