Validation of discrete time-to-event prediction models in the presence of competing risks

Abstract

Clinical prediction models play a key role in risk stratification, therapy assignment and many other fields of medical decision making. Before they can enter clinical practice, their usefulness has to be demonstrated using systematic validation. Methods to assess their predictive performance have been proposed for continuous, binary, and time-to-event outcomes, but the literature on validation methods for discrete time-to-event models with competing risks is sparse. The present paper tries to fill this gap and proposes new methodology to quantify discrimination, calibration, and prediction error (PE) for discrete time-to-event outcomes in the presence of competing risks. In our case study, the goal was to predict the risk of ventilator-associated pneumonia (VAP) attributed to Pseudomonas aeruginosa in intensive care units (ICUs). Competing events are extubation, death, and VAP due to other bacteria. The aim of this application is to validate complex prediction models developed in previous work on more recently available validation data.

Keywords: area under the curve; calibration slope; competing events; discrete time-to-event model; dynamic prediction models; prediction error; validation.

Figure 1

Cause‐specific AUC${}_r(t)$ curves for the three models of interest following Li et al. (2018)

Figure 2

Cause‐specific calibration plots for the different models following Berger and Schmid. The dashed line is the ideal 45 degree line indicating, while the solid lines are simple regression lines

Figure 3

Cause‐specific prediction error curves for the three models of interest, with the number of daily events

Figure 4

Cause‐specific relative error reduction curves for different prediction models