Semiparametric estimation of the relationship between ROC operating points and the test-result scale: application to the proper binormal model

Abstract

Rationale and objectives: Semiparametric methods provide smooth and continuous receiver operating characteristic (ROC) curve fits to ordinal test results and require only that the data follow some unknown monotonic transformation of the model's assumed distributions. The quantitative relationship between cutoff settings or individual test-result values on the data scale and points on the estimated ROC curve is lost in this procedure, however. To recover that relationship in a principled way, we propose a new algorithm for "proper" ROC curves and illustrate it by use of the proper binormal model.

Materials and methods: Several authors have proposed the use of multinomial distributions to fit semiparametric ROC curves by maximum-likelihood estimation. The resulting approach requires nuisance parameters that specify interval probabilities associated with the data, which are used subsequently as a basis for estimating values of the curve parameters of primary interest. In the method described here, we employ those "nuisance" parameters to recover the relationship between any ordinal test-result scale and true-positive fraction, false-positive fraction, and likelihood ratio. Computer simulations based on the proper binormal model were used to evaluate our approach in estimating those relationships and to assess the coverage of its confidence intervals for realistically sized datasets.

Results: In our simulations, the method reliably estimated simple relationships between test-result values and the several ROC quantities.

Conclusion: The proposed approach provides an effective and reliable semiparametric method with which to estimate the relationship between cutoff settings or individual test-result values and corresponding points on the ROC curve.

Figure 1

The receiver operating characteristic (ROC) curves, binormal parameters, and areas under the ROC curve (AUCs) for the three populations used in the simulation studies.

Figure 2

The average behavior of the estimation algorithm (areas under the receiver operating characteristic curve = 0.85 and no skew). (a) Histograms of simulated test-result values (5000 samples of 200 positive and 200 negative cases each) are shown. The negative, positive, and mixed histograms are histograms of the actually negative cases, the actually positive cases, and all cases taken together, respectively. (b,c,d) The results for the average log likelihood ratio, the average true positive fraction and the average false positive fraction, respectively. These averages were taken over bins of test-results, versus the 95% confidence intervals constructed from the average estimated errors and the empirical errors (standard deviation of the distribution of the estimates) are shown also. “Estimated value” here indicates the mean of the samples that belong to that bin.

Figure 3

The average behavior of the estimation algorithm (areas under the receiver operating characteristic curve = 0.85 and high skew). (a) Histograms of simulated test-result values (5000 samples of 200 positive and 200 negative cases each) are shown. The negative, positive, and mixed histograms are those of the actually negative cases, the actually positive cases, and all cases taken together, respectively. (b,c,d) The results for average log likelihood ratio, average true-positive fraction, and average false-positive fraction, respectively. These averages were taken over bins of test-results. Also shown are 95% CIs constructed from the average estimated errors and the empirical errors (standard deviation of the distribution of the estimates).

Figure 4

Results of the regression analysis. Averages were taken over 5000 samples of 100, 200, 400, 600, 800, and 1000 cases each. The 95% confidence intervals constructed from the standard errors are shown also. (a,c,e) The average estimated slope for the no-, medium-, and high-skew ROC curves, respectively. Similarly (b,d,f) depict the average estimated intercept. Results based on ordinary and weighted least squares results are labeled least squares (LSQ) and weighted least squares (WLSQ), respectively. Please note that although the ranges of the vertical axes of Figs 4b, d, and f differ, each spans 1.5 units.

Figure 5

Coverage of the 95% CIs for true-positive fraction (TPF), false-positive fraction (FPF), and log-likelihood ratio (LLR) bins for the no-skew receiver operating characteristic curve shown in Figure 1. The coverage results from our simulation studies involving 200 cases per sample are shown in panels (a), (b) and (c), respectively, whereas those of simulation studies involving 2000 cases per sample are shown in panels (d), (e) and (f).

Figure 6

Coverage of the 95% CIs for true-positive fraction (TPF), false-positive fraction (FPF), and log-likelihood ratio (LLR) bins for the high-skew receiver operating characteristic curve shown in Figure 1. The coverage results from the simulation studies involving 200 cases per sample are shown in (a,b,c), respectively, whereas those of simulation studies involving 2000 cases per sample are shown in (d,e,f).

Figure 7

Output of the automated classifier for benign and malignant cases in ultrasonography, with both model-based and empirical estimates of the thresholds corresponding to true-positive fraction = 0.90 and false-positive fraction = 0.10. Also shown are 95% confidence intervals for the empirical estimates of threshold.

Figure 8

The classifier's model-based-fit and empirical receiver operating characteristic curves together with model-based and empirical 95% confidence intervals for true-positive fraction (TPF) and false-positive fraction (FPF) at thresholds corresponding to a model-based TPF = 0.90 or a model-based FPF = 0.10. One should note that sampling errors are larger near “the center” of the ROC curve due to the sampling properties of proportions, because FPF and TPF are proportions; hence, larger residuals from receiver operating characteristic curve-fitting algorithms must be expected there.