External validation of the 4C Mortality Score for patients with COVID-19 and pre-existing cardiovascular diseases/risk factors

Abstract

Objectives: Predictive algorithms to inform risk management decisions are needed for patients with COVID-19, although the traditional risk scores have not been adequately assessed in Asian patients. We aimed to evaluate the performance of a COVID-19-specific prediction model, the 4C (Coronavirus Clinical Characterisation Consortium) Mortality Score, along with other conventional critical care risk models in Japanese nationwide registry data.

Design: Retrospective cohort study.

Setting and participants: Hospitalised patients with COVID-19 and cardiovascular disease or coronary risk factors from January to May 2020 in 49 hospitals in Japan.

Main outcome measures: Two different types of outcomes, in-hospital mortality and a composite outcome, defined as the need for invasive mechanical ventilation and mortality.

Results: The risk scores for 693 patients were tested by predicting in-hospital mortality for all patients and composite endpoint among those not intubated at baseline (n=659). The number of events was 108 (15.6%) for mortality and 178 (27.0%) for composite endpoints. After missing values were multiply imputed, the performance of the 4C Mortality Score was assessed and compared with three prediction models that have shown good discriminatory ability (RISE UP score, A-DROP score and the Rapid Emergency Medicine Score (REMS)). The area under the receiver operating characteristic curve (AUC) for the 4C Mortality Score was 0.84 (95% CI 0.80 to 0.88) for in-hospital mortality and 0.78 (95% CI 0.74 to 0.81) for the composite endpoint. It showed greater discriminatory ability compared with other scores, except for the RISE UP score, for predicting in-hospital mortality (AUC: 0.82, 95% CI 0.78 to 0.86). Similarly, the 4C Mortality Score showed a positive net reclassification improvement index over the A-DROP and REMS for mortality and over all three scores for the composite endpoint. The 4C Mortality Score model showed good calibration, regardless of outcome.

Conclusions: The 4C Mortality Score performed well in an independent external COVID-19 cohort and may enable appropriate disposition of patients and allocation of medical resources.Trial registration number UMIN000040598.

Keywords: COVID-19; epidemiology; respiratory infections.

Figure 1

Distribution and event rates among scores. Bar plots represent the number of patients across the range of each risk model; (A) 4C mortality score, (B) RISE UP score, (C) A-DROP score, and (D) REMS, respectively. Line plots indicate the rate of outcomes. Orange lines indicate mortality, and green lines indicate composite events. 4C, Coronavirus Clinical Characterisation Consortium; REMS, Rapid Emergency Medicine Score.

Figure 2

Receiver operating characteristic (ROC) curve analysis for in-hospital mortality (A) and for composite outcome (B) in all models. 4C, Coronavirus Clinical Characterisation Consortium; REMS, Rapid Emergency Medicine Score.

Figure 3

Calibration plot of the 4C Mortality Score models for in-hospital mortality (A) and a composite outcome (B). The blue line represents a Loess-smoothed regression line in each calibration plot, being close to the diagonal reference (the dashed line). 4C, Coronavirus Clinical Characterisation Consortium.