AD-VAE: Adversarial Disentangling Variational Autoencoder

Abstract

Face recognition (FR) is a less intrusive biometrics technology with various applications, such as security, surveillance, and access control systems. FR remains challenging, especially when there is only a single image per person as a gallery dataset and when dealing with variations like pose, illumination, and occlusion. Deep learning techniques have shown promising results in recent years using VAE and GAN, with approaches such as patch-VAE, VAE-GAN for 3D Indoor Scene Synthesis, and hybrid VAE-GAN models. However, in Single Sample Per Person Face Recognition (SSPP FR), the challenge of learning robust and discriminative features that preserve the subject's identity persists. To address these issues, we propose a novel framework called AD-VAE, specifically for SSPP FR, using a combination of variational autoencoder (VAE) and Generative Adversarial Network (GAN) techniques. The proposed AD-VAE framework is designed to learn how to build representative identity-preserving prototypes from both controlled and wild datasets, effectively handling variations like pose, illumination, and occlusion. The method uses four networks: an encoder and decoder similar to VAE, a generator that receives the encoder output plus noise to generate an identity-preserving prototype, and a discriminator that operates as a multi-task network. AD-VAE outperforms all tested state-of-the-art face recognition techniques, demonstrating its robustness. The proposed framework achieves superior results on four controlled benchmark datasets-AR, E-YaleB, CAS-PEAL, and FERET-with recognition rates of 84.9%, 94.6%, 94.5%, and 96.0%, respectively, and achieves remarkable performance on the uncontrolled LFW dataset, with a recognition rate of 99.6%. The AD-VAE framework shows promising potential for future research and real-world applications.

Keywords: GAN; face recognition; single sample.

Figure 1

The first part of the proposed AD-VAE, which works as a variational adversarial autoencoder. The $\mathbf{x}$ denotes the image data from $\mathbf{X}$, and ${\mathbf{x}}^{dec}$ denotes the decoder reconstruction from $\mathbf{x}$. The encoder ${\mathbf{E}}_{nc}$ has as input image $\mathbf{x}$ and produces two outputs, the mean ($\mu$) and the log-variance ($\sigma$), which define the parameters of a normal distribution $\mathcal{N}(\mu ,\sigma )$. From distribution $\mathcal{N}(\mu ,\sigma )$, we extract a latent vector $c\sim \mathcal{N}(\mu ,\sigma )$ that serves as input to decoder ${\mathbf{D}}_{ec}$ which outputs the reconstruction ${\mathbf{x}}^{dec}$.

Figure 2

The second part of the proposed architecture of AD-VAE, where $\mathbf{x}$ denotes the image from SSPP data $\mathbf{X}$, ${\mathbf{x}}^{rp}$ denotes the image real prototype $\mathbf{x}$, and $\widehat{\mathbf{x}}$ is the generated prototype from image $\mathbf{x}$. The pre-trained (first part) encoder ${\mathbf{E}}_{nc}$ generates the mean $\mu$ and variation $\sigma$ of $\mathbf{x}$. From distribution $\mathcal{N}(\mu ,\sigma )$, we extract a latent vector $c\sim \mathcal{N}(\mu ,\sigma )$ that concatenates with noise vector $z\sim \mathcal{N}(0,1)$ to serve as the input to generator ${\mathbf{G}}_{en}$ which outputs the prototype $\widehat{\mathbf{x}}$ of $\mathbf{x}$. The discriminator $\mathbf{D}$ (1) determines the id and variation of $\mathbf{x}$; (2) determines the id, variation, and whether $\widehat{\mathbf{x}}$ is real or fake; and (3) determines whether ${\mathbf{x}}_{rp}$ is real or fake.

Figure 3

The prototypes generated by AD-VAE are presented as follows: (a) the sample image with variations, (b) the generated prototype of image (a), and (c) the real prototype of image (a). On the right side, the name of the dataset and the variation are indicated.