All-in-one medical image-to-image translation

Abstract

The growing availability of public multi-domain medical image datasets enables training omnipotent image-to-image (I2I) translation models. However, integrating diverse protocols poses challenges in domain encoding and scalability. Therefore, we propose the "every domain all at once" I2I (EVA-I2I) translation model using DICOM-tag-informed contrastive language-image pre-training (DCLIP). DCLIP maps natural language scan descriptions into a common latent space, offering richer representations than traditional one-hot encoding. We develop the model using seven public datasets with 27,950 scans (3D volumes) for the brain, breast, abdomen, and pelvis. Experimental results show that our EVA-I2I can synthesize every seen domain at once with a single training session and achieve excellent image quality on different I2I translation tasks. Results for downstream applications (e.g., registration, classification, and segmentation) demonstrate that EVA-I2I can be directly applied to domain adaptation on external datasets without fine-tuning and that it also enables the potential for zero-shot domain adaptation for never-before-seen domains.

Keywords: CP: Computational biology; CP: Imaging; contrastive language-image pre-training; image-to-image translation; multi-domain medical image; representation learning; zero-shot domain adaptation.

Graphical abstract

Figure 1

Overview of the proposed EVA-I2I model (A) Integrating domains from different datasets allows training an I2I translation model with strong generalization. (B) Datasets used in this work, where datasets for external validation are in red. (C) The proposed EVA-I2I model transforms the input image to different domains, given prompts. Note that the reference image is an example of the given prompt and is not used for inference.

Figure 2

Axial visualization of translated images generated by comparison methods Cases 1–5 from the in-domain datasets show the transformation from T1 to T2, susceptibility weighted imaging (SWI) to T1, CT to T1, non-fat saturated to fat saturated, and out of phase to in phase. Case 6 and case 7 from the out-domain datasets show the transformation from T2 to T1 and post-contrast dynamic contrast-enhanced (DCE) to pre-contrast DCE. Our EVA-I2I attains better quality and a style that resembles the target image.

Figure 3

Axial visualization of the impact of changing prompts on generated images In case 1, altering the sequence (prompt 1) results in a noticeable change in the appearance of the image. Including additional scan information in prompt 2 has a minimal impact on the appearance. Fat saturation (prompt 3) further suppresses the intensity of fat tissue under the skin, thus rendering the generated image more consistent with the reference image. In case 2, when the specific time point of DCE-MRI is not specified (prompt 1), the intensity enhancement of the cardiac and tumor (red circle) is between the time points before contrast (prompt 2) and the first post-contrast (prompt 3). Note that the proposed model requires only the input image and the target prompt. The reference image is not used as an input but is provided solely for comparison. It corresponds to the given prompt but is from a different subject because the input case lacks the corresponding image of the reference prompt.

Figure 4

Visualization of prompt embedding (A) The t-SNE visualization of prompt embedding space of BioBERT encoding. (B) The t-SNE visualization of prompt embedding space of DCLIP encoding. (C) Bar chart of zero-shot classification on medical imaging characters.

Figure 5

Performance of external domain adaptation (A) The bar chart of brain MRI registration on the HCP Retest dataset. (B) The bar chart of vestibular schwannoma segmentation on the crossMoDA22 dataset. (C) The receiver operating characteristic (ROC) curve of the neoadjuvant therapy (NAT) response prediction on the in-house dataset. Data are represented as mean ± SEM.