Computer-aided classification of breast masses: performance and interobserver variability of expert radiologists versus residents

Abstract

Purpose: To evaluate the interobserver variability in descriptions of breast masses by dedicated breast imagers and radiology residents and determine how any differences in lesion description affect the performance of a computer-aided diagnosis (CAD) computer classification system.

Materials and methods: Institutional review board approval was obtained for this HIPAA-compliant study, and the requirement to obtain informed consent was waived. Images of 50 breast lesions were individually interpreted by seven dedicated breast imagers and 10 radiology residents, yielding 850 lesion interpretations. Lesions were described with use of 11 descriptors from the Breast Imaging Reporting and Data System, and interobserver variability was calculated with the Cohen κ statistic. Those 11 features were selected, along with patient age, and merged together by a linear discriminant analysis (LDA) classification model trained by using 1005 previously existing cases. Variability in the recommendations of the computer model for different observers was also calculated with the Cohen κ statistic.

Results: A significant difference was observed for six lesion features, and radiology residents had greater interobserver variability in their selection of five of the six features than did dedicated breast imagers. The LDA model accurately classified lesions for both sets of observers (area under the receiver operating characteristic curve = 0.94 for residents and 0.96 for dedicated imagers). Sensitivity was maintained at 100% for residents and improved from 98% to 100% for dedicated breast imagers. For residents, the computer model could potentially improve the specificity from 20% to 40% (P < .01) and the κ value from 0.09 to 0.53 (P < .001). For dedicated breast imagers, the computer model could increase the specificity from 34% to 43% (P = .16) and the κ value from 0.21 to 0.61 (P < .001).

Conclusion: Among findings showing a significant difference, there was greater interobserver variability in lesion descriptions among residents; however, an LDA model using data from either dedicated breast imagers or residents yielded a consistently high performance in the differentiation of benign from malignant breast lesions, demonstrating potential for improving specificity and decreasing interobserver variability in biopsy recommendations.

Figure 1:

Receiver operating characteristic (ROC) curve for LDA model training using 12 stepwise selected features for the 1005 training cases. The LDA model accurately classified the lesions, yielding an A_z (AUC) of 0.93. FPF = false-positive fraction, TPF = true-positive fraction.

Figure 2:

Histogram demonstrates the frequency of output values for the LDA training model (x-axis is LDA output, y-axis is frequency). Blue bars represent lesions that were benign at biopsy, and red bars represent lesions that were malignant at biopsy. In general, the LDA model produced lower output values for benign lesions and higher output values for malignant lesions, resulting in good classification performance with an A_z of 0.93.