Comparison of visual grading and free-response ROC analyses for assessment of image-processing algorithms in digital mammography

Abstract

Objective: To compare two methods for assessment of image-processing algorithms in digital mammography: free-response receiver operating characteristic (FROC) for the specific task of microcalcification detection and visual grading analysis (VGA).

Methods: The FROC study was conducted prior to the VGA study reported here. 200 raw data files of low breast density (Breast Imaging-Reporting and Data System I-II) mammograms (Novation DR, Siemens, Germany)-100 of which abnormal-were processed by four image-processing algorithms: Raffaello (IMS, Bologna, Italy), Sigmoid (Sectra, Linköping, Sweden), and OpView v. 2 and v. 1 (Siemens, Erlangen, Germany). Four radiologists assessed the mammograms for the detection of microcalcifications. 8 months after the FROC study, a subset (200) of the 800 images was reinterpreted by the same radiologists, using the VGA methodology in a side-by-side approach. The VGA grading was based on noise, saturation, contrast, sharpness and confidence with the image in terms of normal structures. Ordinal logistic regression was applied; OpView v. 1 was the reference processing algorithm.

Results: In the FROC study all algorithms performed better than OpView v. 1. From the current VGA study and for confidence with the image, Sigmoid and Raffaello were significantly worse (p<0.001) than OpView v. 1; OpView v. 2 was significantly better (p=0.01). For the image quality criteria, results were mixed; Raffaello and Sigmoid for example were better than OpView v. 1 for sharpness and contrast (although not always significantly).

Conclusion: VGA and FROC discordant results should be attributed to the different clinical task addressed.

Advances in knowledge: The method to use for image-processing assessment depends on the clinical task tested.

Figure 1

Calculated figures of merit with the jacknife AFROC (JAFROC) method, for each and for the average observer for the different image-processing algorithms. Error is reported as ±standard deviation.

Figure 2

Graphs (a–f) reporting the observed relative frequency of the scores −5 to 5 of all image-processing comparisons during the VGA experiment. S is the skew value.

Figure 3

Graphs (a–c) reporting the comparison between predicted and observed score frequency for each image-processing comparison with respect to the reference algorithm Siemens OpView v. 1, for the overall image confidence criteria.