Evaluating the effectiveness of abbreviated breast MRI (abMRI) interpretation training for mammogram readers: a multi-centre study assessing diagnostic performance, using an enriched dataset

Abstract

Background: Abbreviated breast MRI (abMRI) is being introduced in breast screening trials and clinical practice, particularly for women with dense breasts. Upscaling abMRI provision requires the workforce of mammogram readers to learn to effectively interpret abMRI. The purpose of this study was to examine the diagnostic accuracy of mammogram readers to interpret abMRI after a single day of standardised small-group training and to compare diagnostic performance of mammogram readers experienced in full-protocol breast MRI (fpMRI) interpretation (Group 1) with that of those without fpMRI interpretation experience (Group 2).

Methods: Mammogram readers were recruited from six NHS Breast Screening Programme sites. Small-group hands-on workstation training was provided, with subsequent prospective, independent, blinded interpretation of an enriched dataset with known outcome. A simplified form of abMRI (first post-contrast subtracted images (FAST MRI), displayed as maximum-intensity projection (MIP) and subtracted slice stack) was used. Per-breast and per-lesion diagnostic accuracy analysis was undertaken, with comparison across groups, and double-reading simulation of a consecutive screening subset.

Results: 37 readers (Group 1: 17, Group 2: 20) completed the reading task of 125 scans (250 breasts) (total = 9250 reads). Overall sensitivity was 86% (95% confidence interval (CI) 84-87%; 1776/2072) and specificity 86% (95%CI 85-86%; 6140/7178). Group 1 showed significantly higher sensitivity (843/952; 89%; 95%CI 86-91%) and higher specificity (2957/3298; 90%; 95%CI 89-91%) than Group 2 (sensitivity = 83%; 95%CI 81-85% (933/1120) p < 0.0001; specificity = 82%; 95%CI 81-83% (3183/3880) p < 0.0001). Inter-reader agreement was higher for Group 1 (kappa = 0.73; 95%CI 0.68-0.79) than for Group 2 (kappa = 0.51; 95%CI 0.45-0.56). Specificity improved for Group 2, from the first 55 cases (81%) to the remaining 70 (83%) (p = 0.02) but not for Group 1 (90-89% p = 0.44), whereas sensitivity remained consistent for both Group 1 (88-89%) and Group 2 (83-84%).

Conclusions: Single-day abMRI interpretation training for mammogram readers achieved an overall diagnostic performance within benchmarks published for fpMRI but was insufficient for diagnostic accuracy of mammogram readers new to breast MRI to match that of experienced fpMRI readers. Novice MRI reader performance improved during the reading task, suggesting that additional training could further narrow this performance gap.

Keywords: Abbreviated breast MRI; Breast cancer; Diagnostic accuracy; Education; FAST MRI; Training.

Fig. 1

CONSORT flow diagram detailing participation in FAST MRI reader training programme

Fig. 2

Bar chart showing the frequency of MRI classifications by whether there was a cancer or not present for each group of readers. a Group 1 b Group 2. Legend: Non-Cancer (Blue filled box), Cancer (Orange filled box)

Fig. 3

Point estimates of accuracy in receiver operating characteristic (ROC) space (sensitivity plotted against 1-specificity) for each reader in the study, coded by group (Group 1 = experienced breast MRI readers, Group 2 = mammogram readers who have undergone a single day’s training to interpret abbreviated breast MRI)

Fig. 4

Sensitivity (95%CI) and specificity (95%CI) for the 1st set of 55 cases compared to 2nd set of 70 cases, demonstrating improvement in per-breast performance (specificity) for Group 2 but not for Group 1

Fig. 5

Box and whisker plot for total time taken (seconds) to report each case The long horizontal blue line represents the median; the top and the bottom of the box represent the 25th and 75th percentiles. The diamond in the box represents the mean. The vertical lines (whiskers) extend to the group minimum and maximum values. The outlier within Group 2 of 10003 seconds has been excluded for this plot. Wilcoxon rank-sum p < 0.0001.

Fig. 6

Per-lesion analysis demonstrated graphically as a jackknife alternative free-response receiver operator characteristics (JAFROC) curve There were 58 biopsy-confirmed cancer lesions in the dataset, equating to a total of 2146 decisions made by the 37 readers. The LLF was overall was 83% (1783/2146); 86% (847/986) for the Group 1 readers and 81% (936/1160) for the Group 2 readers. The reader-averaged weighted JAFROC FOM was 0.93 (95%CI 0.92–0.94) overall. The reader-averaged weighted JAFROC FOM for Group 1 readers of 0.95 (95%CI 0.95–0.96) was significantly higher than for Group 2 (0.91; 95%CI 0.89–0.93); p = 0.004

Fig. 7

Per-woman abbreviated breast MRI (abMRI) analysis a Illustration of the percentage of readers correctly identifying each of the 55 women with breast cancer b Illustration of the percentage of readers correctly identifying each of the 70 women without breast cancer

Fig. 8

a– c: Example cancer case from the dataset. This 25mm diameter Grade 2 carcinoma of no special type (oestrogen receptor positive, progesterone receptor equivocal, Her2 receptor negative and Ki67: 20%) was occult mammographically (a) because it was obscured by mammographically dense fibroglandular tissue. It was seen only subtly on the maximum intensity projection (MIP) image of the FAST MRI (b) as it was partially obscured by background parenchymal enhancement (BPE). However, it is clearly seen as a rim enhancing mass on the FAST MRI stack of slices (c), indicated by a yellow arrow. It was correctly identified as a cancer by 17/17 Group 1 readers and 17/20 Group 2 readers during this study.