Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Mar 16:25:e43110.
doi: 10.2196/43110.

What Does DALL-E 2 Know About Radiology?

Lisa C Adams #  1   2 , Felix Busch #  1 , Daniel Truhn  3 , Marcus R Makowski  4 , Hugo J W L Aerts  5   6 , Keno K Bressem  1   5   6
Affiliations

What Does DALL-E 2 Know About Radiology?

Lisa C Adams et al. J Med Internet Res. .

Abstract

Generative models, such as DALL-E 2 (OpenAI), could represent promising future tools for image generation, augmentation, and manipulation for artificial intelligence research in radiology, provided that these models have sufficient medical domain knowledge. Herein, we show that DALL-E 2 has learned relevant representations of x-ray images, with promising capabilities in terms of zero-shot text-to-image generation of new images, the continuation of an image beyond its original boundaries, and the removal of elements; however, its capabilities for the generation of images with pathological abnormalities (eg, tumors, fractures, and inflammation) or computed tomography, magnetic resonance imaging, or ultrasound images are still limited. The use of generative models for augmenting and generating radiological data thus seems feasible, even if the further fine-tuning and adaptation of these models to their respective domains are required first.

Keywords: DALL-E; artificial intelligence; creating images from text; diagnostic imaging; generative model; image creation; image generation; machine learning; medical imaging; radiology; text-to-image; transformer language model; x-ray.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest: None declared.

Figures

Figure 1
Figure 1
Examples of anatomical structures in x-ray images that were created with DALL-E 2 based on short text descriptions.
Figure 2
Figure 2
Examples of text-to-image–generated anatomical structures in CT, MRI, and ultrasound images created with DALL-E 2. CT: computed tomography; MRI: magnetic resonance imaging.
Figure 3
Figure 3
Reconstructed areas of different anatomical locations in x-rays created by using DALL-E 2. The yellow-bordered regions of the original images were erased before providing the remnant images for reconstruction.
Figure 4
Figure 4
Extending x-ray images of different anatomical regions beyond their borders by using DALL-E 2. The original x-rays are shown in yellow boxes, and the areas outside of the yellow boxes were generated by DALL-E 2.
See this image and copyright information in PMC

References

    1. Sharma M, Savage C, Nair M, Larsson I, Svedberg P, Nygren JM. Artificial intelligence applications in health care practice: Scoping review. J Med Internet Res. 2022 Oct 05;24(10):e40238. doi: 10.2196/40238. https://www.jmir.org/2022/10/e40238/ v24i10e40238 - DOI - PMC - PubMed
    1. Ramesh A, Dhariwal P, Nichol A, Chu C, Chen M. Hierarchical text-conditional image generation with CLIP latents. arXiv. doi: 10.48550/arXiv.2204.06125. Preprint posted online on April 13, 2022 https://arxiv.org/pdf/2204.06125.pdf . - DOI
    1. Kather JN, Laleh NG, Foersch S, Truhn D. Medical domain knowledge in domain-agnostic generative AI. NPJ Digit Med. 2022 Jul 11;5(1):90. doi: 10.1038/s41746-022-00634-5. doi: 10.1038/s41746-022-00634-5.10.1038/s41746-022-00634-5 - DOI - DOI - PMC - PubMed
    1. Conwell C, Ullman T. Testing relational understanding in text-guided image generation. arXiv. doi: 10.48550/arXiv.2208.00005. Preprint posted online on July 29, 2022 https://arxiv.org/pdf/2208.00005.pdf . - DOI
    1. Marcus G, Davis E, Aaronson S. A very preliminary analysis of DALL-E 2. arXiv. doi: 10.48550/arXiv.2204.13807. Preprint posted online on April 25, 2022 https://arxiv.org/vc/arxiv/papers/2204/2204.13807v1.pdf . - DOI

Publication types

LinkOut - more resources