Applications of generative adversarial networks in neuroimaging and clinical neuroscience

Abstract

Generative adversarial networks (GANs) are one powerful type of deep learning models that have been successfully utilized in numerous fields. They belong to the broader family of generative methods, which learn to generate realistic data with a probabilistic model by learning distributions from real samples. In the clinical context, GANs have shown enhanced capabilities in capturing spatially complex, nonlinear, and potentially subtle disease effects compared to traditional generative methods. This review critically appraises the existing literature on the applications of GANs in imaging studies of various neurological conditions, including Alzheimer's disease, brain tumors, brain aging, and multiple sclerosis. We provide an intuitive explanation of various GAN methods for each application and further discuss the main challenges, open questions, and promising future directions of leveraging GANs in neuroimaging. We aim to bridge the gap between advanced deep learning methods and neurology research by highlighting how GANs can be leveraged to support clinical decision making and contribute to a better understanding of the structural and functional patterns of brain diseases.

Keywords: GAN; Generative adversarial network; Neuroimaging; Pathology; Review.

Fig. 1.

Semantics of the original GAN and its extensions. (A) Original GAN architecture where z, x and y denote random noise, generated image, and real image. G and D represent generator and discriminator separately. Wasserstein GAN, deep convolutional GAN, and self-attention GAN share the same structure of the original GAN but use a different loss function, convolutional block and self-attention module, respectively. (B) Progressive growing GAN architecture. The resolution of each generated image x_1,x_2,. .., x_n and real image y_1, y_2,. .., y_n is increasing from left to right. The number of layers within each generator G_1, G_2,. .., G_n and discriminator D_1, D_2,. .., D_n is also growing accordingly. (C) Conditional GAN architecture where z, y, x, x^′ denote random noise, extra label/information, real data, and generated data, respectively. Concatenation of random noise z and label y are input to the generator and concatenation of label y and generated/real data are input to the discriminator. (D) Cycle-GAN structure where x₁, y₂ denote umpired data from two different modalities, and $x_2^{\prime }$, $y_1^{\prime }$ denote generated data for the corresponding modality. G₁ transform data from modality X to Y, and G₂ transforms inversely. Generated data $y_1^{\prime }$ is reconstructed back to input x₁ through G₂ and same for generated data $x_2^{\prime }$. (E) Info-GAN structure, where z, c, x, x^′ denote random noise, informative part of latent variable, real data, and generated data. Concatenation of z and c are input to the generator. Informative latent c are reconstructed through an encoder E from generated data x^′ = G(z, c). (F) MUNIT GAN structure where x₁, y₂, $x_2^{\prime }$, $y_1^{\prime }$ denote unpaired data and generated data from two different modalities, X and Y. $c_{x_1}$, $c_{y_2}$, $s_{x_1}$, $s_{y_2}$ denote content and style variables derived from data from two modalities, respectively. Data x₁ is firstly decoded into content $c_{x_1}$ and style variable $s_{x_1}$ respectively. Then, the content variable $c_{x_1}$ is concatenated with a style variable from the Y modality to generate data $y_1^{\prime }$ through the generator G₁. Concatenation of $c_{x_1}$ and $s_{x_1}$ are used as input for reconstruct x₁ through G₂. Same process also applies to the reverse direction. Images are taken and adapted from Goodfellow et al. (2014), Karras et al. (2018), Chen et al. (2016), Mirza and Osindero (2014), Zhu et al. (2017), Huang et al. (2018).

Fig. 2.

GAN applications in disease classification with single- and multi-modal imaging. (A) Schematic of THS-GAN for Alzheimer’s disease and mild cognitive impairment classifications. (B) Comparison of synthesized brain MR images from THS-GAN and real T1-weighted scans with coronal, sagittal, and axial views for different training epochs. (C) Deviation between real image and synthetic images generated by Rev-GAN. In the deviation image, the yellow color represents large differences, and the dark colors denote small deviations. Images are taken and adapted from Lin et al. (2021), Yu et al. (2021).

Fig. 3.

GAN applications in neuroimaging-based anomaly detection. (A) Schema of AnoGAN. Training is performed on health subjects to learn z, a latent space representing the data distribution. Inference is performed by sampling from z to generate image x^′, such that the difference between x^′ and the anomalous image x_a is minimized. (B) Illustration of the latent space distribution produced by AnoGAN. Images are taken and adapted from Schlegl et al. (2017).

Fig. 4.

GAN applications in modeling healthy brain aging. (A) Schematic of the conditional GAN model for modeling the brain aging process across the whole lifespan. x_i : generator input; ℎ₀ : target age vector; a_d : age difference between current age a_i and target age a₀; ${\widehat{x}}_0$: generator output; v_1, v_2, v^′_1, v^′₂: latent embedding. Generator synthesizes brain image of target age and health state, and judge network gives a discrimination score of whether the image given to the discriminator is real or fake. L_ID, L_rec, L_GAN refer to identity loss, reconstruction loss and adversarial loss, respectively. (B) Examples of healthy brain aging modeling using the GAN described in (A). Bottom panel shows the images synthesized at different target ages a₀, and the top panel shows the absolute difference between input image x_i and synthesized image ${\widehat{x}}_0$. (C) Schematic of the perceptual adversarial network (PGAN). (D) Multi-modal perceptual adversarial network (MPGAN) architecture. x, x^T1, x^T2: input 3D MR volume; G(x), G_T1 (x^T1, x^T2), G_T2 (x^T1, x^T2): generated output; y, y^T1, y^T2: real 3D MR volume; D, D_T1, D_T2: discriminator networks; ϕ: feature extraction network; L_VR, L_P, L_adv refer to voxel-wise reconstruction loss, perceptual loss, and adversarial loss. Images are taken and adapted from Xia et al. (2021), Peng et al. (2021).

Fig. 5.

GAN applications in generating disease progression scans from a single time point. (A) Schematic of DANI net. The input to DANI net is a T1 MR image from subject p at age θ with diagnosis d. The output of the decoder is a set of longitudinal scans. Several loss functions (reconstruction loss L_rec, biological constraints L_bio, discriminator losses L_Dz and L_Db) are combined together to train DANI net using a single time point of subject p. (B) Longitudinal MRIs synthesized using 4D-DANI-Net for a 69 years old cognitive normal subject from three orientations. The blue box indicates the input MRI and other images are synthesized MR scans from the model. Two magnified regions are illustrated at the bottom panel. Images are taken and adapted from Ravi et al. (2019).

Fig. 6.

GAN applications in disease subtypes discovery (four-dimensional coordinate system developed by SMILE-GAN). (A) Voxel-wise statistical comparison (onesided t-test) between cognitive normal subjects and subjects that predominantly belong to each of the four Alzheimer’s disease neuroanatomical patterns. (B) Visualization of subjects that belong to the four subtype clusters in a diamond plot. Images are taken and adapted from Yang et al. (2021).

Fig. 7.

GAN applications in brain lesion (white matter hyperintensities and multiple sclerosis) evolution prediction. (A) Schematic of the GAN for white matter hyperintensities evolution prediction. (B) Disease evolution map examples produced by GAN and the derived irregularity map from two time points. (C) Two-stage conditional GAN for [¹¹ C] PIB PET images generation from multi-sequence MR images for myelin content in multiple sclerosis dynamic prediction. (D) Examples of myelin content changes indicating demyelination (red color) and remyelination (blue color) from both GAN outputs and real PET images. Images are taken and adapted from Wei et al. (2020), Rachmadi et al. (2020).

Fig. 8.

GAN applications in brain tumor growth prediction. (A) GP-GAN architecture for glioma growth prediction. x_gi : generated image at time point i; G_i : generator at time point i; D_i: discriminator at time point i. (B) Growth prediction for subjects with low-grade glioma (left) and high-grade glioma (right) at different time points via GP-GAN. GT: ground truth; Pre: prediction. (C) Schematic of SMIG model. The model is trained to 1) generate an abnormal brain based on a healthy brain from ADNI dataset and tumor volume from TCIA; 2) change tumor location. x_R : image represents a healthy brain or tumor in real location; x_V : tumor volume provided by TCIA; x_G: generated image; G: generator; D: discriminator. (D) SMIG model applications on single patient images from BraTS dataset. Images are taken and adapted from Elazab et al. (2020), Kamli et al. (2020).