Development and validation of the ISARIC 4C Deterioration model for adults hospitalised with COVID-19: a prospective cohort study

Abstract

Background: Prognostic models to predict the risk of clinical deterioration in acute COVID-19 cases are urgently required to inform clinical management decisions.

Methods: We developed and validated a multivariable logistic regression model for in-hospital clinical deterioration (defined as any requirement of ventilatory support or critical care, or death) among consecutively hospitalised adults with highly suspected or confirmed COVID-19 who were prospectively recruited to the International Severe Acute Respiratory and Emerging Infections Consortium Coronavirus Clinical Characterisation Consortium (ISARIC4C) study across 260 hospitals in England, Scotland, and Wales. Candidate predictors that were specified a priori were considered for inclusion in the model on the basis of previous prognostic scores and emerging literature describing routinely measured biomarkers associated with COVID-19 prognosis. We used internal-external cross-validation to evaluate discrimination, calibration, and clinical utility across eight National Health Service (NHS) regions in the development cohort. We further validated the final model in held-out data from an additional NHS region (London).

Findings: 74 944 participants (recruited between Feb 6 and Aug 26, 2020) were included, of whom 31 924 (43·2%) of 73 948 with available outcomes met the composite clinical deterioration outcome. In internal-external cross-validation in the development cohort of 66 705 participants, the selected model (comprising 11 predictors routinely measured at the point of hospital admission) showed consistent discrimination, calibration, and clinical utility across all eight NHS regions. In held-out data from London (n=8239), the model showed a similarly consistent performance (C-statistic 0·77 [95% CI 0·76 to 0·78]; calibration-in-the-large 0·00 [-0·05 to 0·05]); calibration slope 0·96 [0·91 to 1·01]), and greater net benefit than any other reproducible prognostic model.

Interpretation: The 4C Deterioration model has strong potential for clinical utility and generalisability to predict clinical deterioration and inform decision making among adults hospitalised with COVID-19.

Funding: National Institute for Health Research (NIHR), UK Medical Research Council, Wellcome Trust, Department for International Development, Bill & Melinda Gates Foundation, EU Platform for European Preparedness Against (Re-)emerging Epidemics, NIHR Health Protection Research Unit (HPRU) in Emerging and Zoonotic Infections at University of Liverpool, NIHR HPRU in Respiratory Infections at Imperial College London.

Figure 1

Multivariable associations between selected predictors and outcome in final model Variable selection was done in each imputed dataset with backward elimination within each National Health Service region (appendix p 5). Black lines and dots indicate point estimates; red shaded regions and error bars indicate 95% CIs.

Figure 2

Internal–external cross-validation of selected model by National Health Service region Dashed lines indicate lines of perfect calibration-in-the-large (0) and calibration slope (1). Black squares indicate point estimates; bars indicate 95% CIs; diamonds indicate pooled estimates from a random-effects meta-analysis (n=66 705).

Figure 3

Calibration and decision-curve analysis in held-out London region (n=8239) (A) Calibration is shown using locally weighted smoothing (LOESS) across multiply imputed datasets. (B) Net benefit is shown with LOESS for each candidate model compared with the treat-all and treat-none approaches. Points score models are recalibrated to the validation data, resulting in optimistic estimates of net benefit for these models. NEWS2=National Early Warning Score 2. REMS=Rapid Emergency Medicine Score. MEWS=Modified Early Warning Score. qSOFA=quick Sequential Organ Failure Assessment. ACP=Age and C-reactive Protein.

Figure 4

4C Deterioration versus 4C Mortality predictions for London validation cohort (n=8239) and randomly sampled example patients (n=10) 4C Mortality probabilities are calculated from points scores, based on observed mortality risk for each score in the original validation data. (A) Smoothed plot reflects locally estimated scatterplot smoothing fit, stratified by age (<50 years, 50–69 years, or ≥70 years), in the London cohort (n=8239). (B) Ten example patients were randomly sampled from the validation cohort, stratified by deciles of 4C Deterioration model predictions. (C) Characteristics of each example participant, with red indicating characteristics associated with higher risk predictions. M=male. F=female. N=no. Y=yes. RA=breathing room air. O₂=receiving oxygen therapy.