Potential and limitations of machine meta-learning (ensemble) methods for predicting COVID-19 mortality in a large inhospital Brazilian dataset

Abstract

The majority of early prediction scores and methods to predict COVID-19 mortality are bound by methodological flaws and technological limitations (e.g., the use of a single prediction model). Our aim is to provide a thorough comparative study that tackles those methodological issues, considering multiple techniques to build mortality prediction models, including modern machine learning (neural) algorithms and traditional statistical techniques, as well as meta-learning (ensemble) approaches. This study used a dataset from a multicenter cohort of 10,897 adult Brazilian COVID-19 patients, admitted from March/2020 to November/2021, including patients [median age 60 (interquartile range 48-71), 46% women]. We also proposed new original population-based meta-features that have not been devised in the literature. Stacking has shown to achieve the best results reported in the literature for the death prediction task, improving over previous state-of-the-art by more than 46% in Recall for predicting death, with AUROC 0.826 and MacroF1 of 65.4%. The newly proposed meta-features were highly discriminative of death, but fell short in producing large improvements in final prediction performance, demonstrating that we are possibly on the limits of the prediction capabilities that can be achieved with the current set of ML techniques and (meta-)features. Finally, we investigated how the trained models perform on different hospitals, showing that there are indeed large differences in classifier performance between different hospitals, further making the case that errors are produced by factors that cannot be modeled with the current predictors.

Figure 1

Info-gain on the best model features, including populational and classifier output based meta-features. FiO₂: fraction of inspired oxygen; GAM: generalized additive models; KNN: K-nearest neighbors; LightGBM: light gradient boosting machines; pO₂: partial pressure of oxygen; RF: random forest; SVM: support vector machines.

Figure 2

Receiver operating characteristic (ROC) Curve comparing multiple models, trained on the prediction of the death outcome.

Figure 3

A sample decision tree with depth 2, trained on our dataset. At each level but the last, the first line of text in each box shows the variable and its cut before the split.

Figure 4

Mutual information scores on the top-20 base patient features (A) and Pearson correlation scores between the top-20 most predictive features and lethality. Adm: admission; ALT: alanine aminotransferase; AST: aspartate transaminase; FiO₂: fraction of inspired oxygen; TTPA: partial activated thromboplastin time.

Figure 5

Pearson correlation scores between model variables and false positive/negative errors. Adm: admission; FiO₂: fraction of inspired oxygen; INR: international normalized ratio; pCO₂: partial pressure of carbon dioxide.

Figure 6

Comparison of mean normalized values between errors and non-errors. Adm: admission; FiO₂: fraction of inspired oxygen; INR: international normalized ratio; pCO₂: partial pressure of carbon dioxide.

Figure 7

Comparison of ROC curves and AUROC results between different hospitals.

Figure 8

Comparison of ROC curves and AUROC results between different hospitals.

Figure 9

Error rates for each confidence threshold in the Stacking model without populational meta-features (which had the best macro-F1 result). The X-axis shows prediction ranges for the model's confidence score, while the y-axis shows the percentage of hits or misses for the model.