Relationship between Roe and Metz simulation model for multireader diagnostic data and Obuchowski-Rockette model parameters

Abstract

For the typical diagnostic radiology study design, each case (ie, patient) undergoes several diagnostic tests (or modalities) and the resulting images are interpreted by several readers. Often, each reader is asked to assign a confidence-of-disease rating to each case for each test, and the diagnostic tests are compared with respect to reader-performance outcomes that are functions of the reader receiver operating characteristic (ROC) curves, such as the area under the ROC curve. These reader-performance outcomes are frequently analyzed using the Obuchowski and Rockette method, which allows conclusions to generalize to both the reader and case populations. The simulation model proposed by Roe and Metz (RM) in 1997 emulates confidence-of-disease data collected from such studies and has been an important tool for empirically evaluating various reader-performance analysis methods. However, because the RM model parameters are expressed in terms of a continuous decision variable rather than in terms of reader-performance outcomes, it has not been possible to evaluate the realism of the RM model. I derive the relationships between the RM and Obuchowski-Rockette model parameters for the empirical area under the ROC curve reader-performance outcome. These relationships make it possible to evaluate the realism of the RM parameter models and to assess the performance of Obuchowski-Rockette parameter estimates. An example illustrates the application of the relationships for assessing the performance of a proposed upper one-sided confidence bound for the Obuchowski-Rockette test-by-reader variance component, which is useful for sample size estimation.

Keywords: Obuchowski-Rockette; Roe and Metz model; diagnostic radiology; receiver operating characteristic (ROC) curve.

Figure 1

Bias of Obuchowski-Rockette error variance and covariance estimates, computed using the Delong et al [27] method, expressed as a percent of the true error variance ${\sigma }_{\epsilon :\text{OR}}^2$. Outcome is the empirical AUC. Results are computed from data simulated from the constrained nonnull unequal-variance RM model, with 30,000 samples simulated for each of N = 36 input combinations: 3 sample sizes (25+/25−, 50+/50−, 100+,100−) × 4 structures(HL. LL, HH, LH) × 3 nonnull curve pairs (with $A_z^{(1)}$ and $A_z^{\left(2\right)}$ denoting the test 1 and test 2 values.) Ten readers are used for each combination. The parameter values are the same as given in Table 2, except that to produce nonnull simulations the μ₊ values are replaced by μ₊ −0.3 and μ₊ +0.3 for tests 1 and 2 and A_z values replaced by corresponding $A_z^{(1)}=\mathrm{\Phi }\left[\left({\mu }_{+}-0.3\right)/\sqrt{1+b^{-2}}\right]$ and $A_z^{(2)}=\mathrm{\Phi }\left[\left({\mu }_{+}+0.3\right)/\sqrt{1+b^{-2}}\right]$ values, respectively. Bias is computed with respect to the corresponding parameters in Table 3. Error bars indicate corresponding large sample 95% confidence intervals.

Figure 2

Percent bias of Obuchowski-Rockette estimates for the reader variance component ${\sigma }_{R:\text{OR}}^2$, test×reader variance component ${\sigma }_{TR:\text{OR}}^2$, $\mathrm{var}\left({\widehat{\text{AUC}}}_{1•}-{\widehat{\text{AUC}}}_{2•}\right)$ and E [MS (T*R)] based on data simulated from the constrained nonnull unequal-variance RM model as described in Figure 1. Ten readers are used for each combination. The OR estimates are defined by ${\widehat{\sigma }}_{R:\text{OR}}^2=\frac{1}{2}\left[\text{MS}\left(\mathrm{R}\right)-\text{MS}\left(\mathrm{T}\ast \mathrm{R}\right)\right]-{\widehat{\text{Cov}}}_1+{\widehat{\text{Cov}}}_3$; ${\widehat{\sigma }}_{TR:\text{OR}}^2=\text{MS}\left(\text{T∗R}\right)-{\widehat{\sigma }}_{\epsilon }^2+{\widehat{\text{Cov}}}_1+{\widehat{\text{Cov}}}_2-{\widehat{\text{Cov}}}_3$; and $\widehat{\mathrm{var}}\left({\widehat{\text{AUC}}}_1.-{\widehat{\text{AUC}}}_2.\right)=\frac{2}{r}\left[\text{MS}\left(\text{T∗R}\right)+r\left({\widehat{\text{Cov}}}_2-{\widehat{\text{Cov}}}_3\right)\right]$, where MS (R) and MS (T*R) are the OR reader and test×reader mean squares, respectively. Bias is computed with respect to the corresponding parameters in Table 3. Percent bias = 100×bias/ (estimand value).

Figure 3

Observed versus predicted bias for the Obuchowski-Rockette test×reader variance component estimate, ${\widehat{\sigma }}_{\epsilon }^2$, and the variance of the difference of the empirical reader-averaged AUC estimate, $\widehat{\mathrm{var}}\left({\widehat{\text{AUC}}}_1.-{\widehat{\text{AUC}}}_2.\right)$, based on data simulated from the constrained nonnull unequal-variance RM model as described in Figure 1. Observed bias = (mean estimate – estimand), where the estimand is computed from the Table 3 formulas. The predicted bias is a function of the error variance and covariances, as given by (22–23), but with simulation-study estimates of the error-variance and covariance biases used in place of the true bias values.

Figure 4

Empirical coverage of the proposed upper one-sided confidence bound for the Obuchowski-Rockette test-by-reader variance component ${\sigma }_{TR:\text{OR}}^2$, based on data simulated from the constrained nonnull unequal-variance RM model as described in Figure 1, except that there are 108 simulation settings because all three reader levels (3, 5, 10 readers) are included. Empirical coverage is the proportion of samples for a particular input combination such that ${\widehat{\zeta }}_{1-\alpha }\ge {\sigma }_{TR:\text{OR}}^2$, where ${\widehat{\zeta }}_{1-\alpha }$ is the estimated (1 − α) 100% upper one-sided confidence bound and ${\sigma }_{TR:\text{OR}}^2$ is the true value computed from the Table 3 formulas.

Figure 5

Monte Carlo versus predicted variance of the empirical reader-averaged AUC test difference. The Monte Carlo values are based on data simulated from the constrained nonnull unequal-variance RM model as described in Figure 1, except that there are 108 simulation settings because all three reader levels (3, 5, 10 readers) are included. The predicted variance is computed using the Table 3 formula. The dashed line is the fitted linear regression line.