Components-of-variance models and multiple-bootstrap experiments: an alternative method for random-effects, receiver operating characteristic analysis

Abstract

Rationale and objectives: The purpose of this study was to develop an alternative approach to random-effects, receiver operating characteristic analysis inspired by a general formulation of components-of-variance models. The alternative approach is a higher-order generalization of the Dorfman, Berbaum, and Metz (DBM) approach that yields additional information on the variance structure of the problem.

Materials and methods: Six population experiments were designed to determine the six variance components in the DBM model. For practical problems, in which only a finite set of readers and patients are available, six analogous bootstrap experiments may be substituted for the population experiments to estimate the variance components. Monte Carlo simulations were performed on the population experiments, and those results were compared with the corresponding multiple-bootstrap estimates and those obtained with the DBM approach. Confidence intervals on the difference of ROC parameters for competing diagnostic modalities were estimated, and corresponding comparisons were made.

Results: For mean values, the agreement of present estimates of variance structures with population results was excellent and, when suitably weighted and mixed, similar to or closer than that with the DBM method. For many variance structures, the confidence intervals in this study for the difference in ROC area between modalities were comparable to those with the DBM method. When reader variability was large, however, mean confidence intervals from this study were tighter than those with the DBM method and closer to population results.

Conclusion: The jackknife approach of DBM provides a linear approximation to receiver-operating-characteristic statistics that are intrinsically nonlinear. The multiple-bootstrap technique of this study, however, provides a more general, nonparametric, maximum-likelihood approach. It also yields estimates of the variance structure previously unavailable.