Need for objective task-based evaluation of deep learning-based denoising methods: A study in the context of myocardial perfusion SPECT

Abstract

Background: Artificial intelligence-based methods have generated substantial interest in nuclear medicine. An area of significant interest has been the use of deep-learning (DL)-based approaches for denoising images acquired with lower doses, shorter acquisition times, or both. Objective evaluation of these approaches is essential for clinical application.

Purpose: DL-based approaches for denoising nuclear-medicine images have typically been evaluated using fidelity-based figures of merit (FoMs) such as root mean squared error (RMSE) and structural similarity index measure (SSIM). However, these images are acquired for clinical tasks and thus should be evaluated based on their performance in these tasks. Our objectives were to: (1) investigate whether evaluation with these FoMs is consistent with objective clinical-task-based evaluation; (2) provide a theoretical analysis for determining the impact of denoising on signal-detection tasks; and (3) demonstrate the utility of virtual imaging trials (VITs) to evaluate DL-based methods.

Methods: A VIT to evaluate a DL-based method for denoising myocardial perfusion SPECT (MPS) images was conducted. To conduct this evaluation study, we followed the recently published best practices for the evaluation of AI algorithms for nuclear medicine (the RELAINCE guidelines). An anthropomorphic patient population modeling clinically relevant variability was simulated. Projection data for this patient population at normal and low-dose count levels (20%, 15%, 10%, 5%) were generated using well-validated Monte Carlo-based simulations. The images were reconstructed using a 3-D ordered-subsets expectation maximization-based approach. Next, the low-dose images were denoised using a commonly used convolutional neural network-based approach. The impact of DL-based denoising was evaluated using both fidelity-based FoMs and area under the receiver operating characteristic curve (AUC), which quantified performance on the clinical task of detecting perfusion defects in MPS images as obtained using a model observer with anthropomorphic channels. We then provide a mathematical treatment to probe the impact of post-processing operations on signal-detection tasks and use this treatment to analyze the findings of this study.

Results: Based on fidelity-based FoMs, denoising using the considered DL-based method led to significantly superior performance. However, based on ROC analysis, denoising did not improve, and in fact, often degraded detection-task performance. This discordance between fidelity-based FoMs and task-based evaluation was observed at all the low-dose levels and for different cardiac-defect types. Our theoretical analysis revealed that the major reason for this degraded performance was that the denoising method reduced the difference in the means of the reconstructed images and of the channel operator-extracted feature vectors between the defect-absent and defect-present cases.

Conclusions: The results show the discrepancy between the evaluation of DL-based methods with fidelity-based metrics versus the evaluation on clinical tasks. This motivates the need for objective task-based evaluation of DL-based denoising approaches. Further, this study shows how VITs provide a mechanism to conduct such evaluations computationally, in a time and resource-efficient setting, and avoid risks such as radiation dose to the patient. Finally, our theoretical treatment reveals insights into the reasons for the limited performance of the denoising approach and may be used to probe the effect of other post-processing operations on signal-detection tasks.

Keywords: SPECT; deep learning; model observer; task-based evaluation.

FIGURE 1

The workflow of our virtual imaging trial.

FIGURE 2

The workflow to generate the digital-phantom population.

FIGURE 3

The performance of the DL-based denoising approach evaluated using the fidelity-based FoMs and the model-observer study. Each row corresponds to a different defect type. The first two columns show the performance evaluated using RMSE and SSIM, respectively. The third column shows the ROC plots obtained for the normal-dose images and the low-dose image prior to and after conducting the denoising operation at the 10% dose level. The x and y axis of this plot denote the false positive fraction (FPF) and the true positive fraction (TPF), respectively. Finally, the fourth column shows the AUC values at all the considered dose levels.

FIGURE 4

Representative reconstructed images: the upper two rows show examples where the DL-based denoising approach removed the defect (defects marked by yellow arrows). Note that the defect is characterized by a lower uptake than the rest of the myocardium. Lower two rows show examples where the DL-based denoising approach introduced a false defect (marked by red arrows). The normal-dose images (second column) are also shown for reference. Rows 1, 2, and 4 are images from a male patient, and row 3 is from a female patient.

FIGURE 5

The defect regions, intensity profiles of $\text{Δ}\overline{\widehat{f}}$ and $\text{Δ}\overline{v}$ obtained at normal dose and the images at low dose before and after applying the DL-based denoising approach with defect (a) type 1, (b) type 2, (c) type 3, and (d) type 4, at 10% and 5% dose levels. The first three columns show the windowed $\text{Δ}\overline{\widehat{f}}$ for the normal dose, the images at low dose before and after applying the DL-based denoising approach. The fourth column shows the intensity profile of $\text{Δ}\overline{\widehat{f}}$ corresponding to the dashed line in the defect region. The fifth column shows the profile of $\text{Δ}\overline{v}$ for the corresponding $\text{Δ}\overline{\widehat{f}}$.

FIGURE 6

The eigenvalue spectra of the covariance matrix obtained at normal dose and the images at low dose before and after applying the DL-based denoising approach with four types of defects at four dose levels.

FIGURE 7

The observer SNR obtained at normal dose and the images at low dose before and after applying the DL-based denoising approach with four types of defects at four dose levels. Confidence intervals were indicated by black lines.