Integrating the predictiveness of a marker with its performance as a classifier

Abstract

There are two popular statistical approaches to biomarker evaluation. One models the risk of disease (or disease outcome) with, for example, logistic regression. A marker is considered useful if it has a strong effect on risk. The second evaluates classification performance by use of measures such as sensitivity, specificity, predictive values, and receiver operating characteristic curves. There is controversy about which approach is more appropriate. Moreover, the two approaches can give contradictory results on the same data. The authors present a new graphic, the predictiveness curve, which complements the risk modeling approach. It assesses the usefulness of a risk model when applied to the population. Although the predictiveness curve relates to classification performance measures, it also displays essential information about risk that is not displayed by the receiver operating characteristic curve. The authors propose that the predictiveness and classification performance of a marker, displayed together in an integrated plot, provide a comprehensive and cohesive assessment of a risk marker or model. The methods are demonstrated with data on prostate-specific antigen and risk factors from the Prostate Cancer Prevention Trial, 1993-2003.

FIGURE 1.

Predictiveness curve for the risk model shown in table 1 that includes prostate-specific antigen, age, digital rectal examination, and prior biopsy as risk factors for high-grade prostate cancer, Prostate Cancer Prevention Trial, 1993–2003. Shown on right are the risk thresholds of ≥0.20 for high risk and <0.02 for low risk. Open circles display observed proportions of high-grade cancers within risk deciles.

FIGURE 2.

Predictiveness curves for prostate-specific antigen (PSA) alone, PSA and other factors, and the simulated marker (SIM), Prostate Cancer Prevention Trial, 1993–2003.

FIGURE 3.

Schematic diagram showing how classifier performance parameters relate to the predictiveness curve. Positive predictive value = dark/shade dark + intermediate; negative predictive value = white area/white + light shade; true positive fraction = dark shade/dashed box; false positive fraction = intermediate shade/1 − dashed box.

FIGURE 4.

The integrated predictiveness and classification plot for the simulated marker using two criteria for defining a positive biomarker result, Prostate Cancer Prevention Trial, 1993–2003. In part A, the criterion is risk ≥0.20; in part B, the criterion is true positive fraction (TPF) = 0.95. FPF, false positive fraction.

FIGURE 5.

Receiver operating characteristic (ROC) curves for risk models based on prostate-specific antigen (PSA) alone, PSA and other risk factors, and the simulated marker (SIM), Prostate Cancer Prevention Trial, 1993–2003. These ROC curves correspond to the predictiveness curves in figure 2. False positive fraction (FPF) and true positive fraction (TPF) points corresponding to the high-risk designation (risk: ≥0.20) are displayed for each model.