Image Imputation with conditional generative adversarial networks captures clinically relevant imaging features on computed tomography

Abstract

Kidney cancer is among the top 10 most common malignancies in adults, and is commonly evaluated with four-phase computed tomography (CT) imaging. However, the presence of missing or corrupted images remains a significant problem in medical imaging that impairs the detection, diagnosis, and treatment planning of kidney cancer. Deep learning approaches through conditional generative adversarial networks (cGANs) have recently shown technical promise in the task of imputing missing imaging data from these four-phase studies. In this study, we explored the clinical utility of these imputed images. We utilized a cGAN trained on 333 patients, with the task of the cGAN being to impute the image of any phase given the other three phases. We tested the clinical utility on the imputed images of the 37 patients in the test set by manually extracting 21 clinically relevant imaging features and comparing them to their ground truth counterpart. All 13 categorical clinical features had greater than 85% agreement rate between true images and their imputed counterparts. This high accuracy is maintained when stratifying across imaging phases. Imputed images also show good agreement with true images in select radiomic features including mean intensity and enhancement. Imputed images possess the features characteristic of benign or malignant diagnosis at an equivalent rate to true images. In conclusion, imputed images from cGANs have large potential for clinical use due to their ability to retain clinically relevant qualitative and quantitative features.

Fig 1. Project overview schematic.

Each image is imputed by inputting the other three phases into the cGAN model, and both the imputed image and its ground truth counterpart are passed through the clinical feature extraction pipeline. Analysis includes comparing the clinical similarity of the imputed images to their ground truth, as well as comparing each of their abilities to correctly distinguish benign and malignant masses.

Fig 2. Imputed images agree with ground truth (GT) on categorical features.

Bars show the fraction of images which agree with its corresponding image in the GT per feature. (A) All phases, (B) Pre-Contrast Phase, (C) Corticomedullary Phase, (D) Nephrographic phase, (E) Excretory phase.

Fig 3. Imputed images correlate well with their respective ground truth (GT) images on multiple clinically-relevant quantitative features.

(A) Mean intensity, (B) Min intensity, (C) Max intensity, (D) Mean enhancement, (E) Min enhancement, (F) Max enhancement, (G) Lesion size, (H) Lesion size (2nd dimension). Colors = phases; solid line = best fit line; dashed line = y = x.

Fig 4. Imputed images show similar phase intensity diagrams to their ground truth (GT) counterparts.

(A) All 37 imaging studies, split into groups of 9-10 (B-E) for visualization. Color, alpha = imaging study; solid = imputed image; dashed = GT for the corresponding imputed image; Black lines = average values.

Fig 5. Imputed images map clinical features characteristic of benign vs. malignant diagnosis status at a level similar to ground truth (GT).

Bars show the fraction of images which contain the clinical feature that corresponds to the correct diagnosis. (A) All phases, (B) Pre-Contrast Phase, (C) Corticomedullary Phase, (D) Nephrographic phase, (E) Excretory phase.

Fig 6. Standard deviation of imputed images does not correlate with agreement with ground truth (GT).

(A) All phases. (B) Pre-Contrast Phase. (C) Corticomedullary Phase. (D) Nephrographic phase. (E) Excretory phase. (F) Relationship between standard deviation and the fraction of features in agreement with GT per image.

Fig 7. Hallucination Indices for each of the 148 analyzed images.

Gray dashed line = mean (0.3069).

Fig 8. Smooth density plot of radiologist evaluation of similarity between imputed and true images on 1-5 scale.

Black curve = average of four phases; Black dashed line = average score; Gray dashed line = thresholded value for clinical utility and inter-rater variability calculations.