BI-RADS data should not be used to estimate ROC curves

Abstract

After applauding the recent trend of employing receiver operating characteristic (ROC) analysis to measure diagnostic performance in large clinical studies, we discuss why Breast Imaging Reporting and Data System data should not be used to estimate ROC curves in screening mammography.

Graph shows effect of the frequency of screening use of BI-RADS categories 4 and 5 on the empirical ROC curve, based on data in the Table (2). Appendix E1 (online) describes the calculation of these empirical ROC curves. The trapezoidal areas under the empirical ROC curves (AUCs) ± standard errors of the estimate are 0.906 ± 0.004 (green), 0.885 ± 0.004 (red), and 0.876 ± 0.004 (teal). The maximum-likelihood proper binormal AUCs ± standard errors of the estimate are 0.913 ± 0.004 (green), 0.920 ± 0.003 (red), and 0.917 ± 0.004 (teal) (to improve legibility, the smooth ROC curve estimates are not shown). Monte Carlo simulation showed that, because these AUC estimates were obtained by variations in the analysis of a single data set and so were strongly correlated, the statistical significance of their differences was high. FPF = false-positive fraction, TPF = true-positive fraction (subscripted numbers = BI-RADS categories).