Prospective validation of a dynamic prognostic model for identifying COVID-19 patients at high risk of rapid deterioration

Abstract

Background: We sought to develop and prospectively validate a dynamic model that incorporates changes in biomarkers to predict rapid clinical deterioration in patients hospitalized for COVID-19.

Methods: We established a retrospective cohort of hospitalized patients aged ≥18 years with laboratory-confirmed COVID-19 using electronic health records (EHR) from a large integrated care delivery network in Massachusetts including >40 facilities from March to November 2020. A total of 71 factors, including time-varying vital signs and laboratory findings during hospitalization were screened. We used elastic net regression and tree-based scan statistics for variable selection to predict rapid deterioration, defined as progression by two levels of a published severity scale in the next 24 h. The development cohort included the first 70% of patients identified chronologically in calendar time; the latter 30% served as the validation cohort. A cut-off point was estimated to alert clinicians of high risk of imminent clinical deterioration.

Results: Overall, 3706 patients (2587 in the development and 1119 in the validation cohort) met the eligibility criteria with a median of 6 days of follow-up. Twenty-four variables were selected in the final model, including 16 dynamic changes of laboratory results or vital signs. Area under the ROC curve was 0.81 (95% CI, 0.79-0.82) in the development set and 0.74 (95% CI, 0.71-0.78) in the validation set. The model was well calibrated (slope = 0.84 and intercept = -0.07 on the calibration plot in the validation set). The estimated cut-off point, with a positive predictive value of 83%, was 0.78.

Conclusions: Our prospectively validated dynamic prognostic model demonstrated temporal generalizability in a rapidly evolving pandemic and can be used to inform day-to-day treatment and resource allocation decisions based on dynamic changes in biophysiological factors.

Keywords: COVID-19; dynamic model; model development; model validation; prognosis; time-varying.

FIGURE 1

Schematic representation of prognostic dynamic prediction model study design. The nature of the dynamic prediction for severe disease progression is estimating the probability that an individual will experience the occurrence of severe disease progression (Y) within a certain time period (longitudinal relationship showed by the horizontal solid black arrows). In our model, the probability is estimated within 24 h from hospital admission (T = 0) and from each day after hospital admission (T1, T2, etc.). To predict at T0 the probability of the outcome at T1 we used time fixed predictors, while to predict at T1 the probability of the outcome at T2 and onwards we used predictors measured as: A, time fixed predictors (demographics, lifestyle characteristics and comorbidities information) at T0; B, time varying predictors at T1 (T2, T3, etc.); and C, change in time varying predictors between T0 and T1 (T1 and T2 to predict the outcome at T3, T2 and T3 to predict T4, etc.)

FIGURE 2

Area under the curves in development and validation datasets of the dynamic prognostic model predicting severe disease progression in patients hospitalized with COVID‐19. Receiver operating characteristic curves for the development and the validation cohorts of hospitalized patients with COVID‐19 are reported in blue. The grey lines in the plots represent lower limit of a receiver operating characteristic curve where the area under the curve equals 0.5

FIGURE 3

Calibration plot of the agreement between predicted and observed probabilities across deciles of the risk of severe disease progression to examine the presence of over or under prediction in the validation dataset. The figure visualizes the agreement (slope) between predictions and observations in deciles of predicted values (risk of rapid disease progression) in the validation dataset. The x‐axis shows the full range of predicted probabilities (0 to 1), and the y‐axis shows the full range of actual probabilities (0 to 1). The triangles represent the deciles of the predicted values while the solid vertical lines represent the distribution of the probabilities among patients in the validation dataset. The slope = 0.838 is represented by the dashed red line, a slope = 1 is indicative of perfect agreement. A zoom‐in picture has been added to the north‐west side of the figure