Computer-aided diagnosis of lung nodules on CT scans: ROC study of its effect on radiologists' performance

Abstract

Rationale and objectives: The aim of this study was to evaluate the effect of computer-aided diagnosis (CAD) on radiologists' estimates of the likelihood of malignancy of lung nodules on computed tomographic (CT) imaging.

Methods and materials: A total of 256 lung nodules (124 malignant, 132 benign) were retrospectively collected from the thoracic CT scans of 152 patients. An automated CAD system was developed to characterize and provide malignancy ratings for lung nodules on CT volumetric images. An observer study was conducted using receiver-operating characteristic analysis to evaluate the effect of CAD on radiologists' characterization of lung nodules. Six fellowship-trained thoracic radiologists served as readers. The readers rated the likelihood of malignancy on a scale of 0% to 100% and recommended appropriate action first without CAD and then with CAD. The observer ratings were analyzed using the Dorfman-Berbaum-Metz multireader, multicase method.

Results: The CAD system achieved a test area under the receiver-operating characteristic curve (A(z)) of 0.857 +/- 0.023 using the perimeter, two nodule radii measures, two texture features, and two gradient field features. All six radiologists obtained improved performance with CAD. The average A(z) of the radiologists improved significantly (P < .01) from 0.833 (range, 0.817-0.847) to 0.853 (range, 0.834-0.887).

Conclusion: CAD has the potential to increase radiologists' accuracy in assessing the likelihood of malignancy of lung nodules on CT imaging.

Figure 1

Histograms of the longest diameters of the benign and malignant nodules as measured by experienced chest radiologists.

Figure 2

The 10-bin histogram of classifier scores with fitted Gaussian distributions for the malignant and benign classes.

Figure 3

(a) The graphical user interface used by radiologists in the observer study. The first slice of a scan presented is the one containing the nodule marked in a box. (b) The CAD system score that would appear in the upper middle of the screen after the user clicks “Load CAD.”

Figure 4

The averaged ROC curves for the six radiologists without (A_z=0.834) and with CAD (A_z=0.854) (p<0.01) derived from the average slope and intercept parameters of the individual observers’ fitted ROC curves, and the CAD system performance (test A_z=0.857±0.023). CAD, computer-aided diagnosis.

Figure 5

Example of a non-small cell lung cancer that radiologists gave an average likelihood of malignancy of 45.8%, but increased it to 58.3% after seeing the classifier rating of 7, showing the beneficial effect of CAD.

Figure 6

Example of a benign nodule that radiologists gave an average likelihood of malignancy of 53.3%, but was reduced to 48.3% after seeing the classfier score of 4, showing the beneficial effect of CAD.

Figure 7

Biopsy determined that this was adenoid cystic carcinoma, and radiologists gave it an average LM of 57.5%. Though the classifier score of 4 was incorrect, the radiologists changed the likelihood to an average of 55.8%, showing that radiologists are not easily misled by the CAD system if they believe CAD is incorrect.