The Face of Image Reconstruction: Progress, Pitfalls, Prospects

Abstract

Recent research has demonstrated that neural and behavioral data acquired in response to viewing face images can be used to reconstruct the images themselves. However, the theoretical implications, promises, and challenges of this direction of research remain unclear. We evaluate the potential of this research for elucidating the visual representations underlying face recognition. Specifically, we outline complementary and converging accounts of the visual content, the representational structure, and the neural dynamics of face processing. We illustrate how this research addresses fundamental questions in the study of normal and impaired face recognition, and how image reconstruction provides a powerful framework for uncovering face representations, for unifying multiple types of empirical data, and for facilitating both theoretical and methodological progress.

Keywords: face recognition; face space; neural decoding; visual representations.

Figure 1. Facial image reconstruction informs fundamental aspects of face representation.

Notably, reconstruction speaks to the topography of face space and to the separate encoding of shape and surface information. Reconstruction can rely on multiple types of neural and behavioral data as well as directly on image information (e.g., processed by a model or a theoretical observer). An illustration of image reconstruction methodology (adapted with permission from [33]) involves estimating a face space construct from the structure of experimental data, followed by the synthesis of shape and surface features from face space and the combination of such features into image reconstructions of a viewed face (only a representative subset of facial landmarks are displayed for shape; surface features are displayed for L*, a* and b*, which correspond to the lightness, red-green and yellow-blue channels of color vision).

Figure 2. Cortical areas that support identifiable facial image reconstructions across variation in viewpoint and expression.

(a) Sagittal slices showing the location of fMRI-mapped face patches in one monkey along with (b) examples of stimuli and their corresponding reconstructions based on single-unit recordings (SUR) in these areas (adapted with permission from [34]). (c) Axial slices showing the location of cortical areas in the right anterior fusiform gyrus (FG) and the left posterior FG in humans along with (d) examples of stimuli and their corresponding reconstructions based on multivoxel patterns in these areas - additional reconstructions based on behavioral and EEG data are also shown for the same stimuli (adapted with permission from [29] and [33]).

Figure 3. An illustration of deep neural network application to image reconstruction.

(a) An encoder network maps a face image onto latent features, which the generator network converts into a novel face image. The discriminator network evaluates whether a given image, from the original data set or from the generator output, is real or fake (i.e., a generator output). (b) Encoding associates a latent vector with the corresponding brain response pattern. (c) Encoding is inverted to convert a brain response pattern into an estimate of latent face features which the generator network can translate next into a reconstructed face image (adapted from [36]).