Towards better clinical prediction models: seven steps for development and an ABCD for validation

Abstract

Clinical prediction models provide risk estimates for the presence of disease (diagnosis) or an event in the future course of disease (prognosis) for individual patients. Although publications that present and evaluate such models are becoming more frequent, the methodology is often suboptimal. We propose that seven steps should be considered in developing prediction models: (i) consideration of the research question and initial data inspection; (ii) coding of predictors; (iii) model specification; (iv) model estimation; (v) evaluation of model performance; (vi) internal validation; and (vii) model presentation. The validity of a prediction model is ideally assessed in fully independent data, where we propose four key measures to evaluate model performance: calibration-in-the-large, or the model intercept (A); calibration slope (B); discrimination, with a concordance statistic (C); and clinical usefulness, with decision-curve analysis (D). As an application, we develop and validate prediction models for 30-day mortality in patients with an acute myocardial infarction. This illustrates the usefulness of the proposed framework to strengthen the methodological rigour and quality for prediction models in cardiovascular research.

Keywords: Calibration; Clinical usefulness; Discrimination; Missing values; Non-linearity; Prediction model; Shrinkage.

Figure 1

Validation plots for clinical prediction models applied in 17 796 patients enrolled in GUSTO-I outside the USA. The models contained the predictors age (left panel), or age plus Killip class, blood pressure, and heart rate (right panels, with n = 259 or n = 23 034 US patients for model development). (A) calibration-in-the-large calculated as the logistic regression model intercept given that the calibration slope equals 1; (B) calibration slope in a logistic regression model with the linear predictor as the sole predictor; (C) c-statistic indicating discriminative ability. Triangles represent deciles of subjects grouped by similar predicted risk. The distribution of subjects is indicated with spikes at the bottom of the graph, stratified by endpoint (deaths above the x-axis, survivors below the x-axis).

Figure 2

Decision curves for the prediction models applied in 17 796 patients enrolled in GUSTO-I outside the USA. Solid line: Assume no patients are treated, net benefit is zero (no true-positive and no false-positive classifications); Grey line: assume all patients are treated; Dotted lines: patients are treated if predictions exceed a threshold, with 30-day mortality risk predictions based on age only, or a prediction model with age, Killip class, blood pressure, and heart rate, developed in n = 259 or n = 23 034 US patients. The graph gives the expected net benefit per patient relative to no treatment in any patient (‘Treat none’). The threshold defines the weight w for false-positive (treat while patient survived) vs. true-positive (treat a patient who died) classifications. For example, a threshold of 2% implies that FP classifications are valued at 2/98 of true-positive classifications, and w is 0.02 /(1–0.02) = 0.0204. The clinical usefulness of a prediction model can then be summarized as: NB = (TP − w FP)/N, where N is the total number of patients.