Clinical relevance of the ROC and free-response paradigms for comparing imaging system efficacies

Abstract

Observer performance studies are widely used to assess medical imaging systems. Unlike technical/engineering measurements observer performance include the entire imaging chain and the radiologist. However, the widely used receiver operating characteristic (ROC) method ignores lesion localisation information. The free-response ROC (FROC) method uses the location information to appropriately reward or penalise correct or incorrect localisations, respectively. This paper describes a method for improving the clinical relevance of FROC studies. The method consists of assigning appropriate risk values to the different lesions that may be present on a single image. A high-risk lesion is one that is critical to detect and act upon, and is assigned a higher risk value than a low-risk lesion, one that is relatively innocuous. Instead of simply counting the number of lesions that are detected, as is done in conventional FROC analysis, a risk-weighted count is used. This has the advantage of rewarding detections of high-risk lesions commensurately more than detections of lower risk lesions. Simulations were used to demonstrate that the new method, termed case-based analysis, results in a higher figure of merit for an expert who detects more high-risk lesions than a naive observer who detects more low-risk lesions, even though both detect the same total number of lesions. Conventional free-response analysis is unable to distinguish between the two types of observers. This paper also comments on the issue of clinical relevance of ROC analysis vs. FROC for tasks that involve lesion localisation.

Figure 1.

Case-based vs. lesion-based free-response analysis. The upper panel shows case-based AFROC curves for two observers and the lower panel shows corresponding lesion-based curves (i.e. conventional AFROC curves). The low-performance (naive) observer has low detection ability on the high-risk lesions and high detection ability on the low-risk lesions and the expert observer has high detection ability on the high-risk lesions and low detection ability on the low-risk lesions. Simulation parameters were arranged so that both observers detected the same number of lesions. Note the insensitivity of the lesion-based curves to the two observer types. In contrast, in case-based analysis the expert observer curve is clearly above that for the naive observer. Since it is more sensitive to expertise, case-based analysis may be clinically more relevant than lesion-based analysis. [LLF, lesion localisation fraction; FPF, false-positive fraction].