A comparison of modeling approaches for static and dynamic prediction of central-line bloodstream infections using electronic health records (part 1): regression models

Abstract

Background: Hospitals register information in the electronic health records (EHRs) continuously until discharge or death. As such, there is no censoring for in-hospital outcomes. We aimed to compare different static and dynamic regression modeling approaches to predict central line-associated bloodstream infections (CLABSIs) in EHR while accounting for competing events precluding CLABSI.

Methods: We analyzed data from 30,862 catheter episodes at University Hospitals Leuven from 2012 and 2013 to predict 7-day risk of CLABSI. Competing events are discharge and death. Static models using information at catheter onset included logistic, multinomial logistic, Cox, cause-specific hazard, and Fine-Gray regression. Dynamic models updated predictions daily up to 30 days after catheter onset (i.e., landmarks 0 to 30 days) and included landmark supermodel extensions of the static models, separate Fine-Gray models per landmark time, and regularized multi-task learning (RMTL). Model performance was assessed using 100 random 2:1 train-test splits.

Results: The Cox model performed worst of all static models in terms of area under the receiver operating characteristic curve (AUROC) and calibration. Dynamic landmark supermodels reached peak AUROCs between 0.741 and 0.747 at landmark 5. The Cox landmark supermodel had the worst AUROCs (≤ 0.731) and calibration up to landmark 7. Separate Fine-Gray models per landmark performed worst for later landmarks, when the number of patients at risk was low.

Conclusions: Categorical and time-to-event approaches had similar performance in the static and dynamic settings, except Cox models. Ignoring competing risks caused problems for risk prediction in the time-to-event framework (Cox), but not in the categorical framework (logistic regression).

Keywords: Central line–associated bloodstream infection; Dynamic model; Logistic regression; Risk prediction; Survival analysis.

Fig. 1

Frequency of outcomes within 7 days for each of the landmark subsets (LM ≤ 30). The total height of the bar is the number at risk

Fig. 2

Creation of train-test data for estimating performance

Fig. 3

Comparison of performance metrics of dynamic models across landmarks. The vertical Y-axis was truncated for clarity. Minimum mean observed AUROC was 0.535, minimum/maximum mean observed calibration slope was 0.093/1.742, maximum mean observed ECI was 0.386, minimum mean observed scaled BS was −0.133