Evaluating computer-aided detection algorithms

Abstract

Computer-aided detection (CAD) has been attracting extensive research interest during the last two decades. It is recognized that the full potential of CAD can only be realized by improving the performance and robustness of CAD algorithms and this requires good evaluation methodology that would permit CAD designers to optimize their algorithms. Free-response receiver operating characteristic (FROC) curves are widely used to assess CAD performance, however, evaluation rarely proceeds beyond determination of lesion localization fraction (sensitivity) at an arbitrarily selected value of nonlesion localizations (false marks) per image. This work describes a FROC curve fitting procedure that uses a recent model of visual search that serves as a framework for the free-response task. A maximum likelihood procedure for estimating the parameters of the model from free-response data and fitting CAD generated FROC curves was implemented. Procedures were implemented to estimate two figures of merit and associated statistics such as 95% confidence intervals and goodness of fit. One of the figures of merit does not require the arbitrary specification of an operating point at which to evaluate CAD performance. For comparison a related method termed initial detection and candidate analysis was also implemented that is applicable when all suspicious regions are reported. The two methods were tested on seven mammography CAD data sets and both yielded good to excellent fits. The search model approach has the advantage that it can potentially be applied to radiologist generated free-response data where not all suspicious regions are reported, only the ones that are deemed sufficiently suspicious to warrant clinical follow-up. This work represents the first practical application of the search model to an important evaluation problem in diagnostic radiology. Software based on this work is expected to benefit CAD developers working in diverse areas of medical imaging.

Figure 1

This figure illustrates the IDCA approach to fitting FROC operating points. IDCA regards the lesion and non-lesion localization counts as arising from normal and abnormal “cases” in a pseudo-ROC study. The counts are analyzed by ROC curve-fitting software yielding the fitted curve shown in the upper panel. The FROC curve, shown in the lower panel, is obtained by a mapping operation indicated by the arrow, consisting of a point-by-point multiplication of the pseudo-ROC curve along the y-axis by ν′, and along the x-axis by λ′, where (λ′,ν′) are the coordinates of the observed end-point of the FROC curve. Therefore the corner (1, 1) maps to the end-point (λ′,ν′) and each pseudo-ROC point maps to a unique FROC point. Four pseudo-ROC and four FROC operating points and the corresponding cutoffs ζ_i (i = 1, 2, 3, 4) are shown.

Figure 2

This figure compares the cutoffs implicit in the two models; panel (a) corresponds to IDCA and panel (b) corresponds to the search model. For an R-rating free-response study there are R ordered cutoffs ζ_i (i = 1, 2, …, R). The observed non-lesion and lesion localization counts are F_i and T_i, respectively. Since the pseudo-ROC point corresponding to the lowest bin is (1, 1), the corresponding cutoff ζ₁ is negative infinity and no counts are possible below it, whereas in the search model ζ₁ is finite and an unknown number of counts are possible below it. It is this difference that makes search model parameter estimation more difficult.

Figure 3

A typical plot of −LL_λ (λ) vs. λ, where the ordinate is the value of the negative of the log likelihood after it has been minimized with respect to all parameters except λ. It is seen that −LL_λ (λ) has as a minima at λ = λ_M and [λ_L,λ_U ] illustrates the construction of an asymmetric confidence interval for λ, see text.

Figure 4

IDCA (upper panel) and search model (lower panel) fits to data set CAD_A. The dashed curves correspond to the model fits (IDCA – upper panel, or search model – lower panel) and the solid curves are the raw data (the same raw data is plotted in the upper and lower panels). Since the fits are close to the raw data it is difficult to distinguish between them – the “staircase” pattern corresponding to the raw data may be helpful. The solid circles are operating points resulting from binning the data, and the binned data was used by the fitting procedures. They are constrained to lie exactly on the raw curve. Shown are 95% confidence intervals for an intermediate operating point. Both fits are visually excellent although the statistical measures of quality of fit are not as good (p-values 0.0074 and 0.0073, see text for more discussion of this aspect).

Figure 5

As in Figure 4, except this is for data set CAD_B. Both fits are excellent (p-values 0.36 and 0.44).

Figure 6

As in Figure 4, except this is for data set CAD_F. Both fits are excellent (p-values 0.28 and 0.21).