Does deep learning software improve the consistency and performance of radiologists with various levels of experience in assessing bi-parametric prostate MRI?

Abstract

Objective: To investigate whether commercially available deep learning (DL) software improves the Prostate Imaging-Reporting and Data System (PI-RADS) scoring consistency on bi-parametric MRI among radiologists with various levels of experience; to assess whether the DL software improves the performance of the radiologists in identifying clinically significant prostate cancer (csPCa).

Methods: We retrospectively enrolled consecutive men who underwent bi-parametric prostate MRI at a 3 T scanner due to suspicion of PCa. Four radiologists with 2, 3, 5, and > 20 years of experience evaluated the bi-parametric prostate MRI scans with and without the DL software. Whole-mount pathology or MRI/ultrasound fusion-guided biopsy was the reference. The area under the receiver operating curve (AUROC) was calculated for each radiologist with and without the DL software and compared using De Long's test. In addition, the inter-rater agreement was investigated using kappa statistics.

Results: In all, 153 men with a mean age of 63.59 ± 7.56 years (range 53-80) were enrolled in the study. In the study sample, 45 men (29.80%) had clinically significant PCa. During the reading with the DL software, the radiologists changed their initial scores in 1/153 (0.65%), 2/153 (1.3%), 0/153 (0%), and 3/153 (1.9%) of the patients, yielding no significant increase in the AUROC (p > 0.05). Fleiss' kappa scores among the radiologists were 0.39 and 0.40 with and without the DL software (p = 0.56).

Conclusions: The commercially available DL software does not increase the consistency of the bi-parametric PI-RADS scoring or csPCa detection performance of radiologists with varying levels of experience.

Keywords: Deep learning; Magnetic resonance imaging; Prostate cancer.

Fig. 1

The flowchart of the study

Fig. 2

Horizontal bar charts show the PI-RADS scores assigned by the radiologists with (a) and without (b) the DL software

Fig. 3

Cohen’s kappa scores between radiologists without (a) and with (b) the software. There was no statistical difference between the pair-wise kappa scores with and without the DL software

Fig. 4

Radiologists and the DL software in assigning PI-RADS scores. A 64-year-old man with prostate adenocarcinoma with a Gleason Score of 4 + 3 in the right posterolateral peripheral gland at the mid-prostatic level. An axial T2-weighted imaging (a), apparent diffusion coefficient map (b), diffusion-weighted imaging with a high b-value (c), and deep learning decisions overlaid on T2-weighted imaging with a heatmap (d) are shown. The radiologists scored this lesion as PI-RADS 5 with and without the DL software

Fig. 5

Radiologists and the DL software in assigning PI-RADS scores. A 55-year-old man with clinically significant prostate adenocarcinoma with a Gleason Score of 4 + 3 in the left posterior peripheral gland at the basal level. An axial T2-weighted imaging (a), apparent diffusion coefficient map (b), diffusion-weighted imaging with a high b-value (c), and deep learning decisions overlaid on T2-weighted imaging with a heatmap (d) are shown. All radiologists scored PI-RADS 5 for the index lesion. However, the deep learning software failed to identify the index lesion

Fig. 6

The area under the receiver operating curves. The area under the receiver operating curves of the radiologists without (a) and with (b) the deep learning software in identifying clinically significant prostate cancer