GestaltGAN: synthetic photorealistic portraits of individuals with rare genetic disorders

Abstract

The facial gestalt (overall facial morphology) is a characteristic clinical feature in many genetic disorders that is often essential for suspecting and establishing a specific diagnosis. Therefore, publishing images of individuals affected by pathogenic variants in disease-associated genes has been an important part of scientific communication. Furthermore, medical imaging data is also crucial for teaching and training deep-learning models such as GestaltMatcher. However, medical data is often sparsely available, and sharing patient images involves risks related to privacy and re-identification. Therefore, we explored whether generative neural networks can be used to synthesize accurate portraits for rare disorders. We modified a StyleGAN architecture and trained it to produce artificial condition-specific portraits for multiple disorders. In addition, we present a technique that generates a sharp and detailed average patient portrait for a given disorder. We trained our GestaltGAN on the 20 most frequent disorders from the GestaltMatcher database. We used REAL-ESRGAN to increase the resolution of portraits from the training data with low-quality and colorized black-and-white images. To augment the model's understanding of human facial features, an unaffected class was introduced to the training data. We tested the validity of our generated portraits with 63 human experts. Our findings demonstrate the model's proficiency in generating photorealistic portraits that capture the characteristic features of a disorder while preserving patient privacy. Overall, the output from our approach holds promise for various applications, including visualizations for publications and educational materials and augmenting training data for deep learning.

Fig. 1. Images generated by GestaltGAN.

Images in the top row are the latent averages of the disorder, which were generated by averaging the features of the disorder. Images in the bottom row are selected images generated for the respective disorder.

Fig. 2. Visualization for the GestaltGAN architecture.

StyleGAN has been extended by a customized loss function, GestaltLoss, based on the GestaltMatcher ensemble. The conditional generator synthesizes images for 20 different disorders and receives feedback from the discriminator about the origin, which is either artificial or real. For the training of human faces and disorders, data of FFHQ Aging and the GMDB were used. Once the training is finished the generator can be used to synthesize arbitrary amounts of artificial images.

Fig. 3. Comparison of ordinary image averages and latent averages generated for four disorders GestaltGAN was trained on.

Since both averaging techniques operate in essence on the same underlying data, there is a high similarity of image averages (left) and latent averages (right) for each condition. However, averaging in image space blurs fine structures, while latent averages appear more photorealistic.

Fig. 4. The survey was presented to human participants to assess their ability to recognize generated images, specific training images, and specific genetic conditions.

The lower section shows the expected result for each question due to random chance and what was observed. The closer the observed and expected values, the harder the question. 1) Participants could identify original images slightly more often than randomly expected. 2) Participants could not identify which individuals were used for training. 3) Participants could recognize the characteristic features in original and synthetic images with comparable precision. Color code: Original images are depicted in black, original images not part of the training set in yellow, and generated images in blue.