Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2013 Feb 20;32(4):661-72.
doi: 10.1002/sim.5598. Epub 2012 Sep 10.

Predictive accuracy of risk factors and markers: a simulation study of the effect of novel markers on different performance measures for logistic regression models

Peter C Austin  1 , Ewout W Steyerberg
Affiliations
Free PMC article

Predictive accuracy of risk factors and markers: a simulation study of the effect of novel markers on different performance measures for logistic regression models

Peter C Austin et al. Stat Med. .
Free PMC article

Abstract

The change in c-statistic is frequently used to summarize the change in predictive accuracy when a novel risk factor is added to an existing logistic regression model. We explored the relationship between the absolute change in the c-statistic, Brier score, generalized R(2) , and the discrimination slope when a risk factor was added to an existing model in an extensive set of Monte Carlo simulations. The increase in model accuracy due to the inclusion of a novel marker was proportional to both the prevalence of the marker and to the odds ratio relating the marker to the outcome but inversely proportional to the accuracy of the logistic regression model with the marker omitted. We observed greater improvements in model accuracy when the novel risk factor or marker was uncorrelated with the existing predictor variable compared with when the risk factor has a positive correlation with the existing predictor variable. We illustrated these findings by using a study on mortality prediction in patients hospitalized with heart failure. In conclusion, the increase in predictive accuracy by adding a marker should be considered in the context of the accuracy of the initial model.

PubMed Disclaimer

Figures

Figure 1
Figure 1
Relationship between change in c-statistic and c-statistic of univariate model (binary risk factor).
Figure 2
Figure 2
Relationship between change in Brier score and Brier score of univariate model (binary risk factor).
Figure 3
Figure 3
Relationship between change in scaled Brier score and scaled Brier score of univariate model (binary risk factor).
Figure 4
Figure 4
Relationship between change in R2 and R2 of univariate model (binary risk factor).
Figure 5
Figure 5
Relationship between Integrated Discrimination Improvement (IDI) and discrimination slope of univariate model (binary risk factor).
Figure 6
Figure 6
Continuous risk factor.
Figure 7
Figure 7
Multivariable risk model.
Figure 8
Figure 8
Effect of odds ratio and prevalence of risk factor on increase in model accuracy.
See this image and copyright information in PMC

References

    1. Lee DS, Austin PC, Rouleau JL, Liu PP, Naimark D, Tu JV. Predicting mortality among patients hospitalized for heart failure: derivation and validation of a clinical model. Journal of the American Medical Association. 2003;290(19):2581–2587. - PubMed
    1. Lee KL, Woodlief LH, Topol EJ, Weaver WD, Betriu A, Col J, Simoons M, Aylward P, Van de WF, Califf RM. Predictors of 30-day mortality in the era of reperfusion for acute myocardial infarction. Results from an international trial of 41,021 patients. GUSTO-I Investigators. Circulation. 1995;91(6):1659–1668. - PubMed
    1. Roozenbeek B, Maas AI, Lingsma HF, Butcher I, Lu J, Marmarou A, McHugh GS, Weir J, Murray GD, Steyerberg EW. IMPACT Study Group. Baseline characteristics and statistical power in randomized controlled trials: selection, prognostic targeting, or covariate adjustment? Critical Care Medicine. 2009;37:2683–2690. - PubMed
    1. Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. New York, NY: Springer; 2009.
    1. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ, Kattan MW. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128–138. - PMC - PubMed

Publication types

LinkOut - more resources