Development and validation of a reinforcement learning algorithm to dynamically optimize mechanical ventilation in critical care

Abstract

The aim of this work was to develop and evaluate the reinforcement learning algorithm VentAI, which is able to suggest a dynamically optimized mechanical ventilation regime for critically-ill patients. We built, validated and tested its performance on 11,943 events of volume-controlled mechanical ventilation derived from 61,532 distinct ICU admissions and tested it on an independent, secondary dataset (200,859 ICU stays; 25,086 mechanical ventilation events). A patient "data fingerprint" of 44 features was extracted as multidimensional time series in 4-hour time steps. We used a Markov decision process, including a reward system and a Q-learning approach, to find the optimized settings for positive end-expiratory pressure (PEEP), fraction of inspired oxygen (FiO₂) and ideal body weight-adjusted tidal volume (Vt). The observed outcome was in-hospital or 90-day mortality. VentAI reached a significantly increased estimated performance return of 83.3 (primary dataset) and 84.1 (secondary dataset) compared to physicians' standard clinical care (51.1). The number of recommended action changes per mechanically ventilated patient constantly exceeded those of the clinicians. VentAI chose 202.9% more frequently ventilation regimes with lower Vt (5-7.5 mL/kg), but 50.8% less for regimes with higher Vt (7.5-10 mL/kg). VentAI recommended 29.3% more frequently PEEP levels of 5-7 cm H₂O and 53.6% more frequently PEEP levels of 7-9 cmH₂O. VentAI avoided high (>55%) FiO₂ values (59.8% decrease), while preferring the range of 50-55% (140.3% increase). In conclusion, VentAI provides reproducible high performance by dynamically choosing an optimized, individualized ventilation strategy and thus might be of benefit for critically ill patients.

Fig. 1. VentAI Data Routine.

Flow diagram of the overall cohort, architectural overview of the VentAI algorithm and independent testing on eICU dataset.

Fig. 2. VentAI Performance.

a VentAI estimated performance return on both datasets (MIMIC-III and eICU) versus clinicians’ performance return with variance in MIMIC-III dataset after the exposure of the policies to 500 models. b Relation between VentAI performance return and estimated 90-day mortality risk in the MIMIC-III dataset. c Relation between VentAI performance return and in-hospital mortality risk in the eICU dataset.

Fig. 3. Visualization of the action distribution in the 3-dimensional action space (MIMIC-III dataset).

The test set includes 36,225 decision time instances and the designed model facilitates 343 action bins in the action space.

Fig. 4. Number of action changes (MIMIC-III dataset).

The relative number of action changes (ideal body weight-adjusted tidal volume (Vt), positive end expiratory pressure (PEEP), and fraction of inspired oxygen (FiO₂)) is shown in relation to the number of mechanically ventilated patients at each 4 h time step. Clinicians action changes are shown in blue while the VentAI action changes are shown in red.

Fig. 5. Visualization of two representative patient cases (MIMIC-III dataset).

Visualization of two representative case studies in 4-hour intervals. Both patients died within the observed 90 days. Clinicians’ actions are shown in blue while the VentAI actions are shown in red.

Fig. 6. Out-of-Bag feature weight analysis of VentAI (MIMIC-III dataset).

Relative weight of each feature using out-of-bag feature weight analysis, based on the relative loss of prediction, represented by an increase of the mean squared error. a Ideal body weight-adjusted tidal volume (mL/kg). b PEEP (cmH₂0). c FiO₂ (%).