Unifying cardiovascular modelling with deep reinforcement learning for uncertainty aware control of sepsis treatment

Abstract

Sepsis is a potentially life-threatening inflammatory response to infection or severe tissue damage. It has a highly variable clinical course, requiring constant monitoring of the patient's state to guide the management of intravenous fluids and vasopressors, among other interventions. Despite decades of research, there's still debate among experts on optimal treatment. Here, we combine for the first time, distributional deep reinforcement learning with mechanistic physiological models to find personalized sepsis treatment strategies. Our method handles partial observability by leveraging known cardiovascular physiology, introducing a novel physiology-driven recurrent autoencoder, and quantifies the uncertainty of its own results. Moreover, we introduce a framework for uncertainty-aware decision support with humans in the loop. We show that our method learns physiologically explainable, robust policies, that are consistent with clinical knowledge. Further our method consistently identifies high-risk states that lead to death, which could potentially benefit from more frequent vasopressor administration, providing valuable guidance for future research.

Fig 1

Proposed decision support system (A): We use the compete patient history, which includes, vitals, scores, and labs, and previous treatment, to infer hidden states. These would all combine to make the state S_t. Our trained agent, takes this state and outputs value distributions for each treatment, its own uncertainty, and an approximate clinician’s policy. We then factor in all 3 to propose uncertainty-aware treatment strategies. The electrical analog of the cardiovascular model (B) This provides a lumped representation of the resistive and elastic properties of the entire arterial circulation using just two elements, a resistance R and a capacitance C. This model is used to derive algebraic equations relating R, C, stroke volume (SV), filling time (T), to heart rate (F) and pressure. The Cardiac Output (CO) can be then computed as (SV)F. These equations define the decoder of the physiology-driven autoencoder. Complete physiology-driven autoencoder network structure (C) Patient history is sequentially encoded using three neural networks. A patient encoder computes initial cardiovascular state estimates using patient characteristics, a recurrent neural network (RNN) encodes the past history of vitals and scores, up to and including the current time point, and a transition network which takes the previous cardiovascular state, the action and the history representation to output new cardiovascular state estimates.

Fig 2

Reconstruction of two validation patient trajectories using different levels of corruption using the physiology-driven autoencoder, Left: Heart Rate. Right: Systolic Blood Pressure.

Fig 3

Value distributions for validation patients averaged according to different times from death or discharge, Top row: Non Survivors. Bottom row: Survivors.

Fig 4

Scatter plots of scaled features: Top row: Marker colors indicates if ${\widehat{V}}^{*}(S)<5$ (Blue) or ${\widehat{V}}^{*}(S)\ge 5$ (Red) Bottom: Top 10 features measured by feature permutation. Here, l_k denotes the kth component of the latent lab representation.

Fig 5

Top row: Percentage of states with vasopressors recommended for the training and validation states, with time to eventual death. Here a p% voting agent, denotes an agent which only prescribes vasopressors if an only if least p% of the Bootstrapped Ensembles have agree on giving vasopressors. Bottom row: The percentages of states with vasopressors recommended or given with respect to cardiovascular states and SOFA score.

Fig 6

Expected value evolution of the main agent for two patients: (A) A patient who died in the ICU. (B) A survivor. The marker size indicates the parametric uncertainty associated with a particular action. Also shown are the standardized values of SOFA score, Systolic blood pressure, and the unidentifiable cardiovascular state (CO)R. The x-axis indicates the hours from ICU admission. Recommended treatments under various preference parameters: (see Eq 7). (C)(E) Recommendations for the same patient as in (A). (D)(F) Recommendations for the same patient as in (B). Actual clinician treatments: (G) treatment for the patient in (A), (H) treatment for the patient in (B).

Fig 7

(A) Model Uncertainty with time to death for non-survivors, (B) Model Uncertainty with time to discharge for survivors (C) Averaged entropy of value distributions for non-survivors with time to death, (D) Averaged entropy of value distributions for survivors with time to release. (E) Average Model Uncertainty for data points with density less than the p-th percentile.