Time‑dependent ROC curve analysis to determine the predictive capacity of seven clinical scales for mortality in patients with COVID‑19: Study of a hospital cohort with very high mortality

Abstract

Clinical data from hospital admissions are typically utilized to determine the prognostic capacity of Coronavirus disease 2019 (COVID-19) indices. However, as disease status and severity markers evolve over time, time-dependent receiver operating characteristic (ROC) curve analysis becomes more appropriate. The present analysis assessed predictive power for death at various time points throughout patient hospitalization. In a cohort study involving 515 hospitalized patients (General Hospital Number 1 of Mexican Social Security Institute, Colima, Mexico from February 2021 to December 2022) with COVID-19, seven severity indices [Pneumonia Severity Index (PSI) PaO₂/FiO₂ arterial oxygen pressure/fraction of inspired oxygen (Kirby index), the Critical Illness Risk Score (COVID-GRAM), the National Early Warning Score 2 (NEWS-2), the quick Sequential Organ Failure Assessment score (qSOFA), the Fibrosis-4 index (FIB-4) and the Viral Pneumonia Mortality Score (MuLBSTA were evaluated using time-dependent ROC curves. Clinical data were collected at admission and at 2, 4, 6 and 8 days into hospitalization. The study calculated the area under the curve (AUC), sensitivity, specificity, and predictive values for each index at these time points. Mortality was 43.9%. Throughout all time points, NEWS-2 demonstrated the highest predictive power for mortality, as indicated by its AUC values. PSI and COVID-GRAM followed, with predictive power increasing as hospitalization duration progressed. Additionally, NEWS-2 exhibited the highest sensitivity (>96% in all periods) but showed low specificity, which increased from 22.9% at admission to 58.1% by day 8. PSI displayed good predictive capacity from admission to day 6 and excellent predictive power at day 8 and its sensitivity remained >80% throughout all periods, with moderate specificity (70.6-77.3%). COVID-GRAM demonstrated good predictive capacity across all periods, with high sensitivity (84.2-87.3%) but low-to-moderate specificity (61.5-67.6%). The qSOFA index initially had poor predictive power upon admission but improved after 4 days. FIB-4 had a statistically significant predictive capacity in all periods (P=0.001), but with limited clinical value (AUC, 0.639-0.698), and with low sensitivity and specificity. MuLBSTA and IKIRBY exhibited low predictive power at admission and no power after 6 days. In conclusion, in COVID-19 patients with high mortality rates, NEWS-2 and PSI consistently exhibited predictive power for death during hospital stay, with PSI demonstrating the best balance between sensitivity and specificity.

Keywords: COVID-19; ROC curve; hospital mortality; sensitivity; severity of illness index; specificity.

Figure 1

Flowchart of recruitment of hospitalized patients with COVID-19. COVID-19, Coronavirus Disease 2019.

Figure 2

Clinical parameters and treatment of patients with COVID-19 over the first 8 days of hospitalization. (A) Proportion of patients who died or survived. Compared with baseline data, the proportion of patients who died increased significantly on days 7 (P=0.037) and 8 (P=0.007). (B) Proportion of patients in critical condition significantly increased on days 6 (P=0.024), 7 (P=0.002) and 8 (P<0.001). (C) Proportion of patients requiring mechanical significantly increased ventilation on days 3 (P=0.034) and 4-8 (all P<0.001). (D) Proportion of patients with elevated serum D-dimer significantly increased on days 6-8 (all P<0.001). (E) Proportion of patients with elevated serum lactate dehydrogenase significantly increased on days 5 (P=0.016), 6 (P=0.036), 7 (P=0.029) and 8 (P=0.039). (F) Proportion of patients with elevated serum ferritin significantly increased on days 2-8 (all P<0.001). (G) Proportion of patients with elevated blood urea nitrogen significantly increased on days 6 (P=0.011), 7 (P=0.001) and 8 (P=0.006). (H) Proportion of patients requiring antibiotics significantly on days 3 (P=0.014), 4 (P=0.002) and 5-8 (all P<0.001). (I) Proportion of patients requiring amine therapy significantly increased on days 4 (P=0.009), 5 (P=0.002) and 6-8 (all P<0.001). All comparisons were conducted using mixed-effects multinomial logistic regression analysis. ^*P<0.05 vs. baseline. (J) National early warning score 2, (K) pneumonia severity index and (L) COVID-GRAM remained relatively constant over time, although their values differ depending on the reason for discharge from the hospital (improvement or death) ^*P<0.001. COVID-19, Coronavirus Disease 2019; COVID-GRAM, Critical Illness Risk Score.

Figure 3

Changes in predictive power (AUC, with 95% confidence intervals) during hospitalization for (A) National Early Warning Score, (B) Pneumonia Severity Index, (C) COVID-GRAM, (D) MuLBSTA, (E) Quick Sequential Organ Failure Assessment Score, (F) Kirby and (G) fibrosis-4 index. AUC, area under the curve; COVID-GRAM, Critical Illness Risk Score; MuLBSTA, the Viral Pneumonia Mortality Score.