A vision-language foundation model for the generation of realistic chest X-ray images

Abstract

The paucity of high-quality medical imaging datasets could be mitigated by machine learning models that generate compositionally diverse images that faithfully represent medical concepts and pathologies. However, large vision-language models are trained on natural images, and the diversity distribution of the generated images substantially differs from that of medical images. Moreover, medical language involves specific and semantically rich vocabulary. Here we describe a domain-adaptation strategy for large vision-language models that overcomes distributional shifts. Specifically, by leveraging publicly available datasets of chest X-ray images and the corresponding radiology reports, we adapted a latent diffusion model pre-trained on pairs of natural images and text descriptors to generate diverse and visually plausible synthetic chest X-ray images (as confirmed by board-certified radiologists) whose appearance can be controlled with free-form medical text prompts. The domain-adaptation strategy for the text-conditioned synthesis of medical images can be used to augment training datasets and is a viable alternative to the sharing of real medical images for model training and fine-tuning.

Extended Data Fig. 1 |. Intra-prompt diversity of synthetic CXRs.

Three generated samples per prompt are shown, using the prompts “Right-sided pleural effusion” (top row) and “Left-sided pleural effusion” (bottom row). Numbers on the three right columns represent pair-wise MS-SSIM values calculated between each image and the first image.

Extended Data Fig. 2 |. Multi-label classification performance of DenseNet-121 trained on increasing amounts of real, synthetic or a blend of real and synthetic CXRs.

AUROC is macro-averaged across the labels available for all test datasets (Atelectasis, Cardiomegaly, Consolidation, Pleural Effusion, Pneumothorax, Pneumonia, No Finding). Light areas indicate 95% confidence intervals.

Extended Data Fig. 3 |. Effect of additional synthetic training data on multi-label classification performance.

DenseNet-121 was trained on real and varying amounts of synthetic CXRs, which were obtained by sampling RoentGen multiple (2–4) times per prompt, using 1%, 10% and 100% of D_Dev,PA as underlying dataset and text prompt source. The dashed horizontal lines indicate the classification performance of a model trained solely on real CXRs. AUROC is macro-averaged across the labels available for all test datasets (Atelectasis, Cardiomegaly, Consolidation, Pleural Effusion, Pneumothorax, Pneumonia, No Finding). Light areas indicate 95% confidence intervals.

Extended Data Fig. 4 |. Educational use of RoentGen to illustrate signs and pathologies.

Left column, Synthetic CXRs created with the prompts “middle lobe pneumonia” and “right lower lobe pneumonia”, showcasing the “loss of silhouette sign” that can help to distinguish middle lobe pneumonia (silhouette absent) from lower lobe pneumonia (silhouette present) on CXR. Center column, Synthetic CXR showing an opacity at the right hilus, and corresponding diffusion attentive attribution map (DAAM) for the expression “hilar mass”. Right column, Synthetic CXR showing another example of discrete middle lobe pneumonia with an air bronchogram (that is, visible endobronchial air against a background of increased lung opacity, top right image), with the main area of the DAAM highlighting the corresponding opacities in the middle lobe (bottom right image).

Fig. 1 |. Text-to-image synthesis of CXR images using RoentGen, a medical domain-adapted LDM based on the SD pipeline.

A conditional denoising U-Net iteratively denoises a latent vector sampled from a Gaussian distribution over t timesteps. The process is conditioned via cross-attention (QKV for query, key, value of the attention process) through embeddings created from short medical free-text inputs processed by a text encoder E_T. The decoder D of the VAE maps the denoised latent vector to pixel space, resulting in a high-fidelity CXR image showing imaging features corresponding to the initial text prompt.

Fig. 2 |. Text-conditional synthesis of CXRs.

Samples were created by prompting a fine-tuned model (60k training steps; learning rate 5 × 10⁻⁵; PA view) for typical CXR abnormalities. The generated CXRs feature high levels of detail: when prompted for ‘edema’ (top right), perihilar haziness (white arrowheads) and peribronchial cuffing (black arrowhead), both features seen in pulmonary oedema, can be observed. For ‘pneumothorax’ (bottom row, right image), a fine line representing the visceral pleural lining of the partially collapsed lung can be delineated (dashed line). The dotted regions were added for visualization.

Fig. 3 |. Text-conditional appearance of radiological findings.

Here, presence or absence of a finding (pleural effusions, dotted regions of interest added for visualization) and dimensions such as size and laterality were controlled via prompting. Note that the model correctly incorporated the radiological convention of displaying the right patient side on the left side of the image, and vice versa. Each image was picked out of four generated CXRs per respective prompt. Colored text indicates modifiers for size (red), affected side (blue) and negation of a finding (green).

Fig. 4 |. Combination of multiple abnormalities in synthetic CXRs.

The left column contains images created by using the respective caption as prompt, while the middle and right columns highlight findings through an overlay of diffusion attentive attribution maps, which are created by upscaling and aggregating cross-attention word–pixel scores.