Comparative Performance of Deep Learning and Radiologists for the Diagnosis and Localization of Clinically Significant Prostate Cancer at MRI: A Systematic Review

Abstract

Background: Deep learning (DL)-based models have demonstrated an ability to automatically diagnose clinically significant prostate cancer (PCa) on MRI scans and are regularly reported to approach expert performance. The aim of this work was to systematically review the literature comparing deep learning (DL) systems to radiologists in order to evaluate the comparative performance of current state-of-the-art deep learning models and radiologists.

Methods: This systematic review was conducted in accordance with the 2020 Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist. Studies investigating DL models for diagnosing clinically significant (cs) PCa on MRI were included. The quality and risk of bias of each study were assessed using the checklist for AI in medical imaging (CLAIM) and QUADAS-2, respectively. Patient level and lesion-based diagnostic performance were separately evaluated by comparing the sensitivity achieved by DL and radiologists at an identical specificity and the false positives per patient, respectively.

Results: The final selection consisted of eight studies with a combined 7337 patients. The median study quality with CLAIM was 74.1% (IQR: 70.6-77.6). DL achieved an identical patient-level performance to the radiologists for PI-RADS ≥ 3 (both 97.7%, SD = 2.1%). DL had a lower sensitivity for PI-RADS ≥ 4 (84.2% vs. 88.8%, p = 0.43). The sensitivity of DL for lesion localization was also between 2% and 12.5% lower than that of the radiologists.

Conclusions: DL models for the diagnosis of csPCa on MRI appear to approach the performance of experts but currently have a lower sensitivity compared to experienced radiologists. There is a need for studies with larger datasets and for validation on external data.

Keywords: deep learning; magnetic resonance imaging; prostatic neoplasms.

Figure A1

Results of the bias assessment using the QUADAS-2 tool [21,22,23,24,25,26,27,28].

Figure 1

PRISMA 2020 Flow diagram.

Figure 2

An overview of the quality of the included studies evaluated using the CLAIM checklist. The scores reflect the percentage of applicable checklist items that were sufficiently reported by each of the studies. All scores were agreed upon by two reviewers [21,22,23,24,25,26,27,28].

Figure 3

Average accordance with the CLAIM checklist by checklist item. Checklist items that were considered not applicable to the study were omitted from the calculation.

Figure 4

Sensitivity and specificity of DL systems for the diagnosis of csPCa at the patient level, compared to the respective radiologist benchmarks [22,23,24,27,28]. (*) The radiologist’s performance in Hiremath et al. [22] was estimated from ROC curves at specificity estimates derived from the literature [4].