Manual prostate MRI segmentation by readers with different experience: a study of the learning progress

Abstract

Objective: To evaluate the learning progress of less experienced readers in prostate MRI segmentation.

Materials and methods: One hundred bi-parametric prostate MRI scans were retrospectively selected from the Göteborg Prostate Cancer Screening 2 Trial (single center). Nine readers with varying degrees of segmentation experience were involved: one expert radiologist, two experienced radiology residents, two inexperienced radiology residents, and four novices. The task was to segment the whole prostate gland. The expert's segmentations were used as reference. For all other readers except three novices, the 100 MRI scans were divided into five rounds (cases 1-10, 11-25, 26-50, 51-76, 76-100). Three novices segmented only 50 cases (three rounds). After each round, a one-on-one feedback session between the expert and the reader was held, with feedback on systematic errors and potential improvements for the next round. Dice similarity coefficient (DSC) > 0.8 was considered accurate.

Results: Using DSC > 0.8 as the threshold, the novices had a total of 194 accurate segmentations out of 250 (77.6%). The residents had a total of 397/400 (99.2%) accurate segmentations. In round 1, the novices had 19/40 (47.5%) accurate segmentations, in round 2 41/60 (68.3%), and in round 3 84/100 (84.0%) indicating learning progress.

Conclusions: Radiology residents, regardless of prior experience, showed high segmentation accuracy. Novices showed larger interindividual variation and lower segmentation accuracy than radiology residents. To prepare datasets for artificial intelligence (AI) development, employing radiology residents seems safe and provides a good balance between cost-effectiveness and segmentation accuracy. Employing novices should only be considered on an individual basis.

Clinical relevance statement: Employing radiology residents for prostate MRI segmentation seems safe and can potentially reduce the workload of expert radiologists. Employing novices should only be considered on an individual basis.

Key points: • Using less experienced readers for prostate MRI segmentation is cost-effective but may reduce quality. • Radiology residents provided high accuracy segmentations while novices showed large inter-reader variability. • To prepare datasets for AI development, employing radiology residents seems safe and might provide a good balance between cost-effectiveness and segmentation accuracy while novices should only be employed on an individual basis.

Keywords: Artificial intelligence; Learning curve; Magnetic resonance imaging; Prostate cancer.

Fig. 1

T2-weighted axial image from the feedback session in RECOMIA where green color shows segmented areas in agreement, blue color areas segmented by the study participant only, and red color areas segmented by the expert only

Fig. 2

Dice similarity coefficient (DSC) for all readers and all cases with the expert’s segmentations as reference. Feedback sessions were held after cases 10, 25, 50, and 75. DSC above 0.8 was considered accurate (dashed line), the number and percentage of accurate segmentations for each reader are reported. There is high inter-reader variability among novices. Novice 4 showed a clear improvement after round 1 and had only accurate segmentations in subsequent rounds. Novice 3 showed low values in round 2 but then improved to round 3. Novices 1 and 2 had no clear improvement but novice 1 had an overall higher DSC. All residents had consistently high DSC with only three cases of inaccurate segmentations (case 49 for both experienced residents and case 72 for experienced resident 1)

Fig. 3

Boxplot of whole gland segmentation accuracy, with the expert’s segmentations as reference, measured by Dice similarity coefficient (DSC) for readers with different levels of experience and for all rounds. Novices, inexperienced residents, and experienced residents are grouped separately. Novices 1–3 participated in rounds 1–3 (cases 1–50), and all others participated in rounds 1–5 (cases 1–100). Each box covers the data spectrum for a segmentation round and is marked by median, lower, and upper quartiles. Outliers and extreme values are marked with a circle and star respectively and are set according to Tukey’s method. *Only one novice

Fig. 4

Boxplot of segmentation accuracy measured by Dice similarity coefficient per region of the prostate. Novices, inexperienced residents, and experienced residents are grouped separately. Novices 1–3 participated in rounds 1–3 (cases 1–50), and all others participated in rounds 1–5 (cases 1–100). Each box covers the data spectrum and is marked by median, lower, and upper quartiles. Outliers and extreme values are marked with a circle and star respectively and are set according to Tukey’s method