Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Feb-Mar:12035:120350E.
doi: 10.1117/12.2612691. Epub 2022 Apr 4.

Identical-test Roe and Metz simulation model for validating multi-reader methods of analysis for comparing different radiologic imaging modalities

Stephen L Hillis  1
Affiliations

Identical-test Roe and Metz simulation model for validating multi-reader methods of analysis for comparing different radiologic imaging modalities

Stephen L Hillis. Proc SPIE Int Soc Opt Eng. 2022 Feb-Mar.

Abstract

The most frequently used model for simulating multireader multicase (MRMC) data that emulate confidence-of-disease ratings from diagnostic imaging studies has been the Roe and Metz model, proposed by Roe and Metz in 1997 and later generalized by Hillis (2012), Abbey et al (2013) and Gallas and Hillis (2014). All of these models generate continuous confidence-of-disease ratings based on an underlying binormal model for each reader, with the separation between the normal and abnormal rating distributions varying across readers. Numerous studies have used these models for evaluating MRMC analysis and sample size methods. The models suggested in these papers for assessing type I error have been "null" models, where the expected AUC across readers is the same for each test. However, for the null models that have been suggested, there are other differences that would not exist if the two tests were identical. None of the papers cited above discuss how to formulate a null model that is also an "identical-test" model, where the two tests are identical in all respects. The purpose of this paper is to show how to formulate an identical-test model and to discuss the importance of this model. Using the identical-test model, I show through simulations the importance of the Obuchowski-Rockette model constraints to avoid a negative variance estimate, a result which had not previously been empirically demonstrated.

Keywords: Diagnostic radiology; MRMC; Obuchowski and Rockette; ROC curve; Roe and Metz; Type I error; multi-reader multi-case analysis.

PubMed Disclaimer

References

    1. Obuchowski NA and Rockette HE, “Hypothesis testing of diagnostic accuracy for multiple readers and multiple tests: an ANOVA approach with dependent observations,” Communications in Statistics-Simulation and Computation 24(2), 285–308 (1995).
    1. Hillis SL, “A comparison of denominator degrees of freedom methods for multiple observer ROC analysis,” Statistics in Medicine 26(3), 596–619 (2007). - PMC - PubMed
    1. Gallas BD, “One-shot estimate of MRMC variance: AUC,” Academic Radiology 13(3), 353–362 (2006). - PubMed
    1. Gallas BD, Bandos A, Samuelson FW, and Wagner RF, “A framework for random-effects ROC analysis: Biases with the bootstrap and other variance estimators,” Communications in Statistics - Theory and Methods 38(15), 2586–2603 (2009).
    1. Roe CA and Metz CE, “Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data: Validation with computer simulation,” Academic Radiology 4(4), 298–303 (1997). - PubMed

LinkOut - more resources