Impact of computer-aided detection systems on radiologist accuracy with digital mammography

Abstract

Objective: The purpose of this study was to assess the impact of computer-aided detection (CAD) systems on the performance of radiologists with digital mammograms acquired during the Digital Mammographic Imaging Screening Trial (DMIST).

Materials and methods: Only those DMIST cases with proven cancer status by biopsy or 1-year follow-up that had available digital images were included in this multireader, multicase ROC study. Two commercially available CAD systems for digital mammography were used: iCAD SecondLook, version 1.4; and R2 ImageChecker Cenova, version 1.0. Fourteen radiologists interpreted, without and with CAD, a set of 300 cases (150 cancer, 150 benign or normal) on the iCAD SecondLook system, and 15 radiologists interpreted a different set of 300 cases (150 cancer, 150 benign or normal) on the R2 ImageChecker Cenova system.

Results: The average AUC was 0.71 (95% CI, 0.66-0.76) without and 0.72 (95% CI, 0.67-0.77) with the iCAD system (p = 0.07). Similarly, the average AUC was 0.71 (95% CI, 0.66-0.76) without and 0.72 (95% CI 0.67-0.77) with the R2 system (p = 0.08). Sensitivity and specificity differences without and with CAD for both systems also were not significant.

Conclusion: Radiologists in our studies rarely changed their diagnostic decisions after the addition of CAD. The application of CAD had no statistically significant effect on radiologist AUC, sensitivity, or specificity performance with digital mammograms from DMIST.

Keywords: AUC; computer-aided detection; mammography; sensitivity; specificity.

Fig. 1

59-year-old woman with cancerous mass (arrows) missed by majority of readers without and with CAD. A, Right craniocaudal digital mammogram shows cancerous subtle kidney-shaped mass, with iCAD mass mark (circle) adjacent to location of cancer, that was missed by nine of 14 iCAD readers without CAD. Presence of CAD mark changed impression of one reader; remaining eight did not change their initial impression. B, R2 CAD mark (mass area outlined) is overlaid on same mass that was missed by 12 of 15 R2 readers without CAD. Presence of CAD mark did not change any of 12 readers’ impressions. C, High-resolution close-up shows mass.

Fig. 2

49-year-old woman with calcifications. Application of iCAD system resulted in change of reader impression regarding calcifications. Square is computer-aided detection (CAD) true-positive mark denoting cluster of calcifications, and circle is CAD false-positive mark denoting mass. In this cancer case, six of 14 readers missed calcification cluster on initial review without CAD. Once CAD was applied, five of six readers who initially specified no significant findings in right breast changed their impression to significant findings in right breast with probability of malignancy (rating of ≥ 4 on 7-point scale).

Fig. 3

Scatterplots of reader experience versus AUC and sensitivity performance. A and B, Scatterplots graph reader computer-aided detection (CAD) experience against AUC and sensitivity performance with iCAD (A) and R2 (B) systems. C and D, Scatterplots graph reader mammography experience against improvement in AUC and sensitivity performance with iCAD (C) and R2 (D) systems. Dashed horizontal lines at 0.0 indicate no change in performance.