Multireader multicase reader studies with binary agreement data: simulation, analysis, validation, and sizing

Abstract

We treat multireader multicase (MRMC) reader studies for which a reader's diagnostic assessment is converted to binary agreement (1: agree with the truth state, 0: disagree with the truth state). We present a mathematical model for simulating binary MRMC data with a desired correlation structure across readers, cases, and two modalities, assuming the expected probability of agreement is equal for the two modalities ([Formula: see text]). This model can be used to validate the coverage probabilities of 95% confidence intervals (of [Formula: see text], [Formula: see text], or [Formula: see text] when [Formula: see text]), validate the type I error of a superiority hypothesis test, and size a noninferiority hypothesis test (which assumes [Formula: see text]). To illustrate the utility of our simulation model, we adapt the Obuchowski-Rockette-Hillis (ORH) method for the analysis of MRMC binary agreement data. Moreover, we use our simulation model to validate the ORH method for binary data and to illustrate sizing in a noninferiority setting. Our software package is publicly available on the Google code project hosting site for use in simulation, analysis, validation, and sizing of MRMC reader studies with binary agreement data.

Keywords: Monte Carlo simulation; binary data; multireader multicase; reader study; sizing.

Fig. 1

An overview of the procedure for simulating binary MRMC data (additional input parameters include the number of readers $N_r$ and the number of cases $N_c$).

Fig. 2

Simulation results for the validation of the ORH method in terms of the coverage probability of 95% confidence intervals (CIs) under various experimental conditions. The standard error of the estimated coverage probability is $\sqrt{0.95\times 0.05/10000}=0.002$ (error bar not shown for clarity). (L for low and H for high relative magnitude of certain correlations. The first letter represents the between-case correlation. The second letter represents the between-reader correlation. The third letter represents the difference between within-modality correlation and the between-modality correlation).