An interpretable RL framework for pre-deployment modeling in ICU hypotension management

Abstract

Computational methods from reinforcement learning have shown promise in inferring treatment strategies for hypotension management and other clinical decision-making challenges. Unfortunately, the resulting models are often difficult for clinicians to interpret, making clinical inspection and validation of these computationally derived strategies challenging in advance of deployment. In this work, we develop a general framework for identifying succinct sets of clinical contexts in which clinicians make very different treatment choices, tracing the effects of those choices, and inferring a set of recommendations for those specific contexts. By focusing on these few key decision points, our framework produces succinct, interpretable treatment strategies that can each be easily visualized and verified by clinical experts. This interrogation process allows clinicians to leverage the model's use of historical data in tandem with their own expertise to determine which recommendations are worth investigating further e.g. at the bedside. We demonstrate the value of this approach via application to hypotension management in the ICU, an area with critical implications for patient outcomes that lacks data-driven individualized treatment strategies; that said, our framework has broad implications on how to use computational methods to assist with decision-making challenges on a wide range of clinical domains.

Fig. 1. The decision point pipeline.

a We optimize a similarity metric to reflect physicians' perceptions of patient similarity. b We identify decision regions (DR), or areas where similar patients frequently receive different treatments. c We summarize patient trajectories in terms of decision regions. d We use this Markov Decision Process to learn an optimal treatment policy for each decision region.

Fig. 2. Probability assigned to each action under policies.

It is obtained using three different reward functions, compared to the current practices from clinicians.

Fig. 3. The feature means for points in each decision region.

Means are sorted by descending order of Mean Arterial Pressure (MAP) value.

Fig. 4. Change in patient state after different treatments.

a The probability of patients moving from one decision region to other decision regions when given different treatments. This example figure shows how patients transition out of decision region 9. b The expected average feature value for patients in each decision region after different treatments were given. This example figure reflects MAP level change.

Fig. 5. Examples of patient trajectories.

For three specific patient cases where the recommended action differs from the clinician actions, we track the changes over time in their clinical metrics.

Fig. 6. Feature importance of clinical variables.

Bar charts showing feature importance, derived from random forest classifiers, in descending order.