Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Comparative Study
. 2012 Dec;19(12):1508-17.
doi: 10.1016/j.acra.2012.09.012.

Multi-reader ROC studies with split-plot designs: a comparison of statistical methods

Nancy A Obuchowski  1 , Brandon D Gallas, Stephen L Hillis
Affiliations
Comparative Study

Multi-reader ROC studies with split-plot designs: a comparison of statistical methods

Nancy A Obuchowski et al. Acad Radiol. 2012 Dec.

Abstract

Rationale and objectives: Multireader imaging trials often use a factorial design, in which study patients undergo testing with all imaging modalities and readers interpret the results of all tests for all patients. A drawback of this design is the large number of interpretations required of each reader. Split-plot designs have been proposed as an alternative, in which one or a subset of readers interprets all images of a sample of patients, while other readers interpret the images of other samples of patients. In this paper, the authors compare three methods of analysis for the split-plot design.

Materials and methods: Three statistical methods are presented: the Obuchowski-Rockette method modified for the split-plot design, a newly proposed marginal-mean analysis-of-variance approach, and an extension of the three-sample U-statistic method. A simulation study using the Roe-Metz model was performed to compare the type I error rate, power, and confidence interval coverage of the three test statistics.

Results: The type I error rates for all three methods are close to the nominal level but tend to be slightly conservative. The statistical power is nearly identical for the three methods. The coverage of 95% confidence intervals falls close to the nominal coverage for small and large sample sizes.

Conclusions: The split-plot multireader, multicase study design can be statistically efficient compared to the factorial design, reducing the number of interpretations required per reader. Three methods of analysis, shown to have nominal type I error rates, similar power, and nominal confidence interval coverage, are available for this study design.

PubMed Disclaimer

Figures

Figure 1
Figure 1
Type I error rates of three methods: marginal mean ANOVA test statistic (mm ANOVA) (Equation 6) plotted with circles, modified OR test statistic (Equation 3) plotted with squares, and three-sample U-statistic (equation 10) plotted with diamonds. The nominal type I error rate was 0.05.
Figure 2
Figure 2
Coverage of 95% Confidence Intervals of three methods: marginal mean ANOVA test statistic (circles), modified OR test statistic (squares), and three-sample U-statistic (diamonds).
See this image and copyright information in PMC

References

    1. Metz CE. Basic principles of ROC analysis. Semin Nucl Med. 1978;8:283–298. - PubMed
    1. Zweig MH, Campbell G. Receiver operating characteristic plots: a fundamental evaluation tool in clinical medicine. Clin Chem. 1993;39:561–577. - PubMed
    1. Pepe MS. The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford University Press; 2004.
    1. Zhou XH, Obuchowski NA, McClish DL. Statistical Methods in Diagnostic Medicine. 2. Wiley and Sons, Inc; New York: 2011.
    1. Wagner RF, Metz CE, Campbell G. Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol. 2007:723–748. - PubMed

Publication types