Data-Driven Volumetric Image Generation from Surface Structures using a Patient-Specific Deep Leaning Model

Abstract

The advent of computed tomography significantly improves patients' health regarding diagnosis, prognosis, and treatment planning and verification. However, tomographic imaging escalates concomitant radiation doses to patients, inducing potential secondary cancer by 4%. We demonstrate the feasibility of a data-driven approach to synthesize volumetric images using patients' surface images, which can be obtained from a zero-dose surface imaging system. This study includes 500 computed tomography (CT) image sets from 50 patients. Compared to the ground truth CT, the synthetic images result in the evaluation metric values of 26.9 ± 4.1 Hounsfield units, 39.1 ± 1.0 dB, and 0.965 ± 0.011 regarding the mean absolute error, peak signal-to-noise ratio, and structural similarity index measure. This approach provides a data integration solution that can potentially enable real-time imaging, which is free of radiation-induced risk and could be applied to image-guided medical procedures.

Fig. 1.

Volumetric image reconstruction using x-ray projections and non-ionizing radiation surface images. a, Mechanical- and deep learning-based image reconstruction schemes in the context of radiation dose and data quantity. b, Volumetric image reconstruction using a surface image.

Fig. 2.

Model hierarchy for the surface-to-volume network architecture. Model hierarchy of the deep learning networks including three primary components. The reconstruction network generates volumetric images from the surface image. The verification network manufactures a surface from the 3D synthetic CT to compare with the measured surface. The refinement network conserves the material attenuation characteristics between the ground truth CT images and synthetic CT images. The numbers denote the kernel sizes used in each layer within each network

Fig. 3.

Evaluate the performance of the surface-to-volume image reconstruction network. Violin plot to show the distributions of each evaluation metrics including mean absolute error (MAE), peak signal-to-noise ratio (PSNR), and structural similarity index measure (SSIM). Groups are classified using k-mean clustering with inputs from the three-evaluation metric.

Fig. 4.

Examples of predicted volumetric images from each group. The transversal views of ground truth and predicted CT are displayed. The evaluation metrics include structural similarity index measure (SSIM), relative difference maps, and line profiles. The horizontal solid and oblique dot lines on the ground truth images indicate the location of profile comparisons. The training, validation, and testing datasets for each case include 1280, 160, and 160 CT images.

Fig. 5. —

Examples of predicted volumetric images from each group with histogram distributions of CT numbers. The sagittal views of ground truth and predicted CT are displayed. The training, validation, and testing datasets for each case include 1280, 160, and 160 CT images.

Fig. 6. —

Analysis of generated volumetric images. a, The t-distributed stochastic neighbor embedding (t-SNE) visualization of the k-mean clustering results of the total generated volumetric images based on the evaluation metrics of MAE, PSNR, and SSIM. b, Violin plot to show the coefficient of variance (COV) distributions of surface curvature for input data (surface images). c, Surface curvature distributions for patient body structures sampled from each clustered group.