External validation of a Cox prognostic model: principles and methods

Abstract

Background: A prognostic model should not enter clinical practice unless it has been demonstrated that it performs a useful role. External validation denotes evaluation of model performance in a sample independent of that used to develop the model. Unlike for logistic regression models, external validation of Cox models is sparsely treated in the literature. Successful validation of a model means achieving satisfactory discrimination and calibration (prediction accuracy) in the validation sample. Validating Cox models is not straightforward because event probabilities are estimated relative to an unspecified baseline function.

Methods: We describe statistical approaches to external validation of a published Cox model according to the level of published information, specifically (1) the prognostic index only, (2) the prognostic index together with Kaplan-Meier curves for risk groups, and (3) the first two plus the baseline survival curve (the estimated survival function at the mean prognostic index across the sample). The most challenging task, requiring level 3 information, is assessing calibration, for which we suggest a method of approximating the baseline survival function.

Results: We apply the methods to two comparable datasets in primary breast cancer, treating one as derivation and the other as validation sample. Results are presented for discrimination and calibration. We demonstrate plots of survival probabilities that can assist model evaluation.

Conclusions: Our validation methods are applicable to a wide range of prognostic studies and provide researchers with a toolkit for external validation of a published Cox model.

Figure 1

Histogram of the PI in the derivation and validation datasets. The PI was centered on the mean in the derivation dataset. The vertical lines show the 16th, 50th and 84th centiles of the PI in each dataset.

Figure 2

Kaplan-Meier curves for recurrence-free survival in 4 risk groups in the derivation and validation datasets, based on an MFP model.

Figure 3

Baseline cumulative hazard function in the derivation dataset. Jagged curve, empirical (Kaplan-Meier-like) estimate; smooth line and grey band, FP2 fit with pointwise 95% confidence interval determined by bootstrap resampling.

Figure 4

Calibration of survival probabilities in the validation dataset. Smooth lines: recurrence-free survival as predicted in the derivation and validation datasets from the PI and the smoothed baseline survival function from the derivation dataset. Jagged lines: Kaplan-Meier estimates in the three risk groups in the validation dataset. Note that the pairs of smooth curves for the two highest risk groups happen nearly to coincide and are visually indistinguishable.

Figure 5

Empirical cumulative distribution functions of the PI by dataset and risk group. Solid lines, derivation data; dashed lines, validation data.

Figure 6

Estimates of the baseline survival function in the validation dataset. The grey region is a pointwise 95% confidence interval based on bootstrap resampling. Set text for details.