Developing and externally validating machine learning models to forecast short-term risk of ventilator-associated pneumonia

Abstract

Purpose: Ventilator-associated pneumonia (VAP) remains one of the most serious hospital-acquired infections in the intensive care unit (ICU), with high morbidity and mortality. Early identification of patients at risk for developing VAP could enable timely diagnostics and intervention. However, current clinical tools are limited in their ability to detect early physiologic signals preceding VAP onset. We aimed to build supervised machine learning models to predict short term onset of VAP.

Methods: We analyzed electronic health record data from a prospective observational cohort of ICU patients, where VAP was adjudicated using a standardized published protocol by a panel of critical care physicians. Clinical features (including vital signs, ventilator settings, laboratory values, and support devices) were extracted for each patient-ICU-day. We explored unsupervised clustering to characterize feature dynamics associated with VAP onset. We built multiple machine learning models across different prediction windows (3, 5, 7 days before VAP). We examined model performance in two external cohorts, MIMIC-IV and secondary analysis of the AMIKINHAL trial. Results were evaluated with discrimination metrics such as AUROC.

Results: The internal cohort included 507 patients with BAL-confirmed diagnoses: 261 developed VAP and 246 did not have VAP. Visualization using clustering identified distinct physiologic states enriched for VAP-labeled days. The best-performing model achieved an AUROC of 0.866 in predicting VAP up to seven days before clinical diagnosis. Temporal model probability trajectories showed rising model confidence in the days leading up to VAP. On external validation in MIMIC-IV, the best model achieved an AUROC of 0.817 for forecasting VAP within five days. There was low feature overlap with the AMIKINHAL trial data, leading to poor model performance. Feature analysis revealed that platelet count, positive end-expiratory pressure (PEEP), ventilator duration, and inflammatory markers were key drivers of model predictions.

Conclusions: Machine learning models trained on routinely collected ICU data with careful labeling can anticipate VAP onset up to a week in advance with strong predictive performance. Model performance generalized to data from an entirely different hospital system despite differences in practice and labeling patterns, but did not perform well when there was poor feature overlap. Future work should focus on real-time prospective evaluation.

Keywords: machine learning; mechanical ventilation; ventilator-associated pneumonia.

Figure 1:. Overview of the machine learning pipeline for ventilator-associated pneumonia (VAP) onset prediction.

(A) Cohort derivation and labeling from BAL-confirmed VAP or non-VAP cases with 3-, 5-, and 7-day windows. (B) Data preprocessing with feature extraction, normalization, and missingness handling. (C) Model development using logistic regression, random forest, and XGBoost with Optuna tuning. (D) Temporal splitting and evaluation: 10-fold patient-grouped cross-validation on training set with hyperparameter optimization. Temporal train/test split (90/10; test = most recent patients). Within training, 10-fold GroupKFold CV prevents patient leakage. Each fold optimizes hyperparameters (25 Optuna trials) and evaluates on fold validation data. The hyperparameter set achieving the highest mean validation AUC across all folds is selected, and a final model is trained on all training data using these selected hyperparameters and evaluated on the temporal test set.

Figure 2:. Unsupervised clustering reveals physiologic states associated with impending VAP.

(A) Similarity-based clustering of patient-days over a 5-day prediction window identifies distinct physiologic clusters, with VAP-labeled days enriched in clusters marked by heightened inflammation and increased ventilatory support. (B) Aggregated temporal trajectories of key clinical variables in the five days preceding BAL-confirmed VAP demonstrate coordinated shifts across inflammatory, hemodynamic, and metabolic features. (C) UMAP projection of patient-day feature vectors shows spatial concentration of VAP days within specific regions of the embedded space, despite pneumonia labels not being used during clustering.

Figure 3:. Model interpretability and temporal risk dynamics for VAP prediction.

(A) SHAP summary plot for the XGBoost model predicting VAP within five days, illustrating the direction and magnitude of feature contributions across patient-days. Higher values of ventilatory support variables (e.g., days on ventilator, plateau pressure) and impaired neurologic status were associated with increased predicted VAP risk. (B) Feature importance rankings from a Random Forest model trained on the same prediction task, demonstrating similar prioritization. (C) Temporal evolution of predicted VAP risk probabilities from Day −5 to Day −1 prior to diagnosis.