Real-time prediction of COVID-19 related mortality using electronic health records

Abstract

Coronavirus disease 2019 (COVID-19) is a respiratory disease with rapid human-to-human transmission caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Due to the exponential growth of infections, identifying patients with the highest mortality risk early is critical to enable effective intervention and prioritisation of care. Here, we present the COVID-19 early warning system (CovEWS), a risk scoring system for assessing COVID-19 related mortality risk that we developed using data amounting to a total of over 2863 years of observation time from a cohort of 66 430 patients seen at over 69 healthcare institutions. On an external cohort of 5005 patients, CovEWS predicts mortality from 78.8% (95% confidence interval [CI]: 76.0, 84.7%) to 69.4% (95% CI: 57.6, 75.2%) specificity at sensitivities greater than 95% between, respectively, 1 and 192 h prior to mortality events. CovEWS could enable earlier intervention, and may therefore help in preventing or mitigating COVID-19 related mortality.

Fig. 1. A selected number of clinical risk factors, and corresponding SOFA, modified early warning score (MEWS) and CovEWS scores for two contrasting patient timelines.

Positive (red) and negative (blue) importance contributions (coloured areas above the clinical time series, see Section “Feature Importance”) indicate to what degree the risk factor at that time point contributed to increasing or decreasing to the mortality risk predicted by CovEWS, respectively. a Patient A’s oxygen saturation (SPO₂) fluctuates significantly before dropping below 95% after around 150 h since her COVID-19 diagnosis, suggesting respiratory distress. The patient is subsequently intubated. This is followed by a sharp rise in serum creatinine levels, indicating potential acute kidney injury. Both SOFA and CovEWS reflect these events with an increase in Patient A’s risk. Crucially, however, since CovEWS accounts for early deterioration in SPO₂ and white blood cell counts, it identifies the patient as high-risk much sooner than SOFA, triggering re-evaluation of current treatment strategy, including investigation for delayed complication or treatment injury, and/or the initiation of goals of care discussion. b In Patient B, different risk factors, including c-reactive protein (CRP), respiratory rate (RR) and SPO₂, weigh heavily in risk assessment. Initially, Patient B’s RR increases significantly to over 30 breaths per minute while her SPO₂ drops below 95%, reflected by a corresponding increase in both SOFA and CovEWS. Patient B’s RR and CRP levels however stabilise, which is correctly reflected in a lowering of the mortality risk by CovEWS. Intubation is averted for this patient. In contrast, SOFA does not account for the improvements in SPO₂, RR and does not reflect Patient B’s improved state.

Fig. 2. Performance comparison in terms of Specificity at greater than either 90% (topmost row) or 95% (other rows) Sensitivity (y-axis) for different prediction horizons ahead of observed mortality events (in hours, x-axis) for CovEWS (light green), CovEWS (linear; light purple), Liang et al. (orange), COVID-19 Estimated Risk for Fatality (COVER_F; blue), Sequential Organ Failure Assessment (SOFA; green), Modified Early Warning Score (MEWS; turquoise), and Yan et al. (red) on the held-out Optum test set, the external TriNetX test set, and selected patient subgroups from the Optum test set.

Some methods do not reach 90% and 95% sensitivity for some horizons, and may therefore not be visible in all plots. Bars indicate median and error bars indicate 95% confidence intervals (CIs) obtained via bootstrapping with 200 samples. Detailed results are available in "Performance Evaluation’’. One-sided Mann-Whitney-Wilcoxon tests were used to derive p values shown at the top of each plot for superiority of CovEWS over CovEWS [linear].

Fig. 3. Stratified survival analysis.

Stratification of patients in a. the held-out Optum test cohort (left, 14,215 patients) and b. the external TriNetX test cohort (right, 5005 patients) according to their assigned CovEWS score over time (in hours since COVID-19 diagnosis) into those patients that were assigned a CovEWS score below 60 (orange, bottommost), from 60 to 69 (deep blue), 70 to 79 (green), 80 to 89 (turquoise), and 90 to 100 (red, topmost). Shaded areas indicate 95% CIs calculated on the logarithmic scale from the standard errors of the Kaplan–Meier estimator with the centre values corresponding to the the Kaplan–Meier estimates. Note that the five strata and their respective limits were chosen for clarity of visualisation—other strata are possible, and may, depending on context, have better clinical utility. Rows show time-varying survival probabilities (top row), the number of patients (centre row), and the cumulative number of mortality events observed (bottom row) for patients in each stratum of assigned CovEWS scores. Steeper curves indicate that more patients died while assigned a CovEWS score in the respective stratum. In contrast to traditional survival curves, cohorts as defined by strata of CovEWS scores are not static over time, and patients move between the stratified groups as they are assigned lower or higher CovEWS scores in response to their status improving or deteriorating, respectively. The results showed that CovEWS enables effective stratification of patients into risk groups over the course of their disease, as patients that were assigned a higher CovEWS score were more likely to die over time on both test cohorts while maintaining separation between the stratified cohorts.