The predictive receiver operating characteristic curve for the joint assessment of the positive and negative predictive values

Abstract

Binary test outcomes typically result from dichotomizing a continuous test variable, observable or latent. The effect of the threshold for test positivity on test sensitivity and specificity has been studied extensively in receiver operating characteristic (ROC) analysis. However, considerably less attention has been given to the study of the effect of the positivity threshold on the predictive value of a test. In this paper we present methods for the joint study of the positive (PPV) and negative predictive values (NPV) of diagnostic tests. We define the predictive receiver operating characteristic (PROC) curve that consists of all possible pairs of PPV and NPV as the threshold for test positivity varies. Unlike the simple trade-off between sensitivity and specificity exhibited in the ROC curve, the PROC curve displays what is often a complex interplay between PPV and NPV as the positivity threshold changes. We study the monotonicity and other geometric properties of the PROC curve and propose summary measures for the predictive performance of tests. We also formulate and discuss regression models for the estimation of the effects of covariates.

Figure 1

Predictive curves with a=0.8. (a) b=0.7, (b) b=1, (c) b=1.5. Solid line, high prevalence (p=0.7); dot-dashed line, low prevalence (p=0.3).

Figure 2

Each non-monotone PROC curve consists of three monotone curve segments defined on $(-\infty ,c_{\text{PPV}}^{*})$, $[c_{\text{PPV}}^{*},c_{\text{NPV}}^{*}]$ and $(c_{\text{NPV}}^{*},\infty )$, respectively. Circles denote points corresponding to operating thresholds at −1 and a+b, triangles denote points corresponding to operating thresholds at −2 and a+2b and crosses denote $c_{\text{PPV}}^{*}$ and $c_{\text{NPV}}^{*}$. (a) a=1, b=0.5, p=0.5; (b) a=2, b=2, p=0.3.

Figure 3

The movement of the PROC curves with increasing a's. (a) b=0.7, (b) b=1 and (c) b=1.5. Solid line, a=5; dotted line, a=2; dot-dashed line, a=0.5.

Figure 4

Each PROC curve segment corresponds to $c\in [-3,a+3b]$, where the prevalence is 0.3. (a) a=0.8, (b) a=2. Solid line, b=0.9; dotted line, b=0.95; dot-dashed line, b=1; dashed line, b=1.05; triple dot-dashed line, b=1.1.

Figure 5

The empirical predictive performance of SUV-lean. (a) Positive predictive value; (b) negative predictive value; (c) PROC curve.

Figure 6

The estimated PROC curve of SUV-lean (dot-dashed line). The circles indicate the pairs of empirical PPV and NPV.

Figure 7

The estimated distance function of SUV-lean.

Figure 8

The empirical (solid line) and estimated (dot-dashed line) PROC curves of digital and screen-film mammography from three readers. The circles represent the pairs of empirical PPV and NPV. The number of participants, the prevalence and the estimates of parameters a and b are also provided. (a) Reader 1 film (N=118, p=0.398, a=1.749, b=1.537); (b) reader 2 film (N=119, p=0.395, a=1.882, b=1.927); (c) reader 3 film (N=94, p=0.269, a=1.231, b=0.918); (d) reader 3 digital (N=94, p=0.269, a=1.031, b=1.044).

Figure 9

Pseudo-likelihood analysis of mammography data. (a) The estimated latent distributions of the diseased and non-diseased subjects under the binormal assumption with estimated operating thresholds c1, …, c5, (b) the estimated positive predictive value, (c) the estimated negative predictive value and (d) the estimated distance to the perfect prediction with 95% CIs. (a) Dotted line, non-diseased; solid line, diseased film; dot-dashed line, diseased digital. (b–d) Solid line, film; dot-dashed line, digital.

Figure 10

The estimated partial PROC curves of film (solid line) and digital mammography (dot-dashed line).