Machine learning models compared with current clinical indices to predict the outcome of high flow nasal cannula therapy in acute hypoxemic respiratory failure

Abstract

Background: Early identification of patients with acute hypoxemic respiratory failure (AHRF) who are at risk of failing high-flow nasal cannula (HFNC) therapy could facilitate closer monitoring, and timely adjustment/escalation of treatment. We aimed to establish whether machine learning (ML) models could predict HFNC outcome, early in the course of treatment, with greater accuracy than currently used clinical indices.

Methods: We developed ML models trained using measurements made within the first 2 h of treatment from 184 AHRF patients (37% HFNC failures) treated at the respiratory ICU of the University Hospital of Modena between 2018 and 2023. For external validation, we used a dataset on 567 AHRF patients (22% failures) comprising 510 patients from the recent RENOVATE trial in Brazil and 57 from the MIMIC-IV and eICU databases in the US. Predictive performance of the ML models was benchmarked against optimized thresholds of the following clinical indices: respiratory rate oxygenation index (ROX) and variants, heart rate to saturation of pulse oxygen (SpO₂) ratio, SpO₂/FiO₂ ratio, PaO₂/FiO₂ ratio, sequential organ failure assessment and heart rate, acidosis, consciousness, oxygenation and respiratory rate scores.

Results: Internal and external predictive performance of a Support Vector Machine (SVM) ML model was superior to all clinical indices across all scenarios tested. In external validation on the 567-patient dataset, a SVM model trained on non-invasive measurements had an accuracy of 73%, sensitivity of 73%, specificity of 73%, and AUC of 0.79. The ROX index had an accuracy of 64%, sensitivity of 79%, specificity of 60%, and AUC of 0.74. When arterial blood gasses (ABG's) were also used for model training, the SVM model had an accuracy of 83%, sensitivity of 84%, specificity of 82%, and AUC of 0.82 in external validation on the MIMIC-IV/eICU dataset. The modified ROX index, which requires PaO₂, achieved 70% accuracy, 63% sensitivity, 74% specificity, and AUC of 0.65.

Conclusions: Decision support tools based on SVM models could provide clinicians with more accurate early predictions of HFNC outcome than currently available clinical indices. If available, ABG measurements could improve the capability to accurately identify patients at risk of failing HFNC therapy.

Fig. 1

Flow chart of the data extraction process from RENOVATE and MIMIC-IV/eICU Databases. a Study cohort extracted from RENOVATE study b Study cohort extracted from eICU/MIMIC-IV database

Fig. 2

Process for model development, validation, performance analysis, and deployment

Fig. 3

Graphical comparison of the performance of both machine learning models with the ROX index and its variants, on the external validation cohort. a, b The Receiver Operating Characteristic (ROC) curve comparing the performance of various machine learning models and the ROX index, along with its variants, on the external validation cohort. c, d Decision curve analysis (DCA) for SVM model and ROX with its variants on the external validation cohort. The curves demonstrate the net benefit across a range of threshold probabilities, comparing the predictive performance of each model against the ‘Treat-All’ (escalate oxygenation treatment for all patients) and ‘Treat-None’ (do not escalate oxygenation treatment) strategies. The shaded regions represent the range of net benefit for SVM. Net benefit = (true positives/total patients)—(false positives/total patients) × $\omega$, where $\omega$ is the odds at the threshold probability. Left Panels: ROC and DCA curves show the performance of a SVM model without using ABG measurements—because such measurements were unavailable in the RENOVATE dataset at matching time points—along with ROX indices. The evaluation was conducted on the combined eICU, MIMIC‑IV, and RENOVATE dataset, which includes 567 patients (440 successes and 127 failures). Right Panels: ROC and DCA curves comparing the performance of a SVM model with all measurements, a SVM model without using ABG measurements, and ROX indices with variants on the eICU + MIMIC-IV dataset (57 patients: 38 successes, 19 failures)

Fig. 4

SVM model explainabilty analysis: SHAP summary plot and permutation importance plot. a, c Average SHAP values obtained from 500 iterations of repeated fivefold cross-validation on internal dataset. Horizontal axis: The impact of each feature on the model’s prediction. Positive SHAP values indicate that the feature contributes to predicting a HFNC failure, while negative SHAP values indicate a contribution to predicting HFNC success. Vertical axis: The list of features used for making a prediction, ordered based on their importance, with the most important features at the top. Dots: Each dot represents a single patient in the dataset. Colour gradient: Indicates the feature value for each observation, where blue represents low feature values and red represents high feature values. b, d Permutation importance for external validation obtained by SVM model on external dataset. The suffix “_diff” indicates the change in a specific variable between two time points. For instance, “FiO2_diff” represents the change in FiO2 values between T1 and T0 (FiO2_diff = FiO2_T1—FiO2_T0)