Binary and multi-category ratings in a laboratory observer performance study: a comparison

Abstract

The authors investigated radiologists, performances during retrospective interpretation of screening mammograms when using a binary decision whether to recall a woman for additional procedures or not and compared it with their receiver operating characteristic (ROC) type performance curves using a semi-continuous rating scale. Under an Institutional Review Board approved protocol nine experienced radiologists independently rated an enriched set of 155 examinations that they had not personally read in the clinic, mixed with other enriched sets of examinations that they had individually read in the clinic, using both a screening BI-RADS rating scale (recall/not recall) and a semi-continuous ROC type rating scale (0 to 100). The vertical distance, namely the difference in sensitivity levels at the same specificity levels, between the empirical ROC curve and the binary operating point were computed for each reader. The vertical distance averaged over all readers was used to assess the proximity of the performance levels under the binary and ROC-type rating scale. There does not appear to be any systematic tendency of the readers towards a better performance when using either of the two rating approaches, namely four readers performed better using the semi-continuous rating scale, four readers performed better with the binary scale, and one reader had the point exactly on the empirical ROC curve. Only one of the nine readers had a binary "operating point" that was statistically distant from the same reader's empirical ROC curve. Reader-specific differences ranged from -0.046 to 0.128 with an average width of the corresponding 95% confidence intervals of 0.2 and p-values ranging for individual readers from 0.050 to 0.966. On average, radiologists performed similarly when using the two rating scales in that the average distance between the run in individual reader's binary operating point and their ROC curve was close to zero. The 95% confidence interval for the fixed-reader average (0.016) was (-0.0206, 0.0631) (two-sided p-value 0.35). In conclusion the authors found that in retrospective observer performance studies the use of a binary response or a semi-continuous rating scale led to consistent results in terms of performance as measured by sensitivity-specificity operating points.

Figure 1

Reader-specific empirical ROC curves (dashed curves), binary operating points (small dots), average binary operating point (large dot), and the empirical ROC curve for the pooled set of ratings. Each operating point (small dot) is connected to the corresponding point on the empirical ROC curve with a vertical line. The lengths of the vertical segments correspond to the absolute value of the distances (∣TPF₀-TPF∣_FPF0∣). A point being above the related ROC curve (the vertical segment extending downward from the point) results in a positive value of the signed distance (TPF₀-TPF∣_FPF0>0). The solid curve depicts the empirical ROC curve for the pooled ratings of all readers. This curve is shown solely for illustration purposes and was not used in the actual analysis. The vertical distance between the average binary operating point and the pooled ROC curve is 0.0093 and the average of the vertical distances actually used in the analysis is 0.0159.