Optimal Vasopressin Initiation in Septic Shock: The OVISS Reinforcement Learning Study

Abstract

Importance: Norepinephrine is the first-line vasopressor for patients with septic shock. When and whether a second agent, such as vasopressin, should be added is unknown.

Objective: To derive and validate a reinforcement learning model to determine the optimal initiation rule for vasopressin in adult, critically ill patients receiving norepinephrine for septic shock.

Design, setting, and participants: Reinforcement learning was used to generate the optimal rule for vasopressin initiation to improve short-term and hospital outcomes, using electronic health record data from 3608 patients who met the Sepsis-3 shock criteria at 5 California hospitals from 2012 to 2023. The rule was evaluated in 628 patients from the California dataset and 3 external datasets comprising 10 217 patients from 227 US hospitals, using weighted importance sampling and pooled logistic regression with inverse probability weighting.

Exposures: Clinical, laboratory, and treatment variables grouped hourly for 120 hours in the electronic health record.

Main outcome and measure: The primary outcome was in-hospital mortality.

Results: The derivation cohort (n = 3608) included 2075 men (57%) and had a median (IQR) age of 63 (56-70) years and Sequential Organ Failure Assessment (SOFA) score at shock onset of 5 (3-7 [range, 0-24, with higher scores associated with greater mortality]). The validation cohorts (n = 10 217) were 56% male (n = 5743) with a median (IQR) age of 67 (57-75) years and a SOFA score of 6 (4-9). In validation data, the model suggested vasopressin initiation in more patients (87% vs 31%), earlier relative to shock onset (median [IQR], 4 [1-8] vs 5 [1-14] hours), and at lower norepinephrine doses (median [IQR], 0.20 [0.08-0.45] vs 0.37 [0.17-0.69] µg/kg/min) compared with clinicians' actions. The rule was associated with a larger expected reward in validation data compared with clinician actions (weighted importance sampling difference, 31 [95% CI, 15-52]). The adjusted odds of hospital mortality were lower if vasopressin initiation was similar to the rule compared with different (odds ratio, 0.81 [95% CI, 0.73-0.91]), a finding consistent across external validation sets.

Conclusions and relevance: In adult patients with septic shock receiving norepinephrine, the use of vasopressin was variable. A reinforcement learning model developed and validated in several observational datasets recommended more frequent and earlier use of vasopressin than average care patterns and was associated with reduced mortality.

Figure 1.. Patient Accrual in a Study of a Reinforcement Learning Rule for Vasopressin Initiation in Septic Shock

eICU-CRD indicates eICU Collaborative Research Database; MIMIC-IV, Medical Information Mart for Intensive Care; UCSF, University of California, San Francisco; UPMC, University of Pittsburgh Medical Center.

Figure 2.. Comparison of Clinician-Observed Administration of Vasopressin With Treatment Recommended by the Reinforcement Learning Rule

A and B, 100 randomly selected patients included in each panel. Each line represents 1 patient trajectory. Red indicates the patient received norepinephrine alone, with a color scale representing the norepinephrine dose, and purple indicates both norepinephrine and vasopressin were infused. White corresponds to discharge alive. Black boxes at the end of the trajectory represent mortality. C, Number of patients in whom vasopressin was initiated in each time block for the clinical observed actions. D, Number of patients in whom vasopressin was initiated per the reinforcement learning rule.

Figure 3.. Weighted Importance Sampling

Weighted importance sampling measures the mean individual reward obtained using the reinforcement learning rule and the mean reward associated with the clinician-observed actions. Weighted importance sampling was estimated in the internal and external validation sets (overall reward) for each reward component independently (reward component) and each internal and external validation set separately (internal/external validation). Results are presented as the difference in weighted importance sampling between the reinforcement learning rule and the clinician’s observed rule, with bootstrapped 95% CIs. A negative weighted importance sampling difference indicated that the clinician-observed actions were associated with a higher reward, whereas a positive difference suggested the reinforcement learning rule yielded a higher reward. For example, the reinforcement learning rule was associated with a higher overall reward (weighted importance sampling difference, 31 [95% CI, 15-52]) as well as higher rewards for each component individually. In the UCSF internal validation set, the lower bound of the 95% CI crossed 0 (weighted importance sampling difference, 15 [95% CI, −48 to 129]), indicating that the overall reward obtained with the reinforcement learning rule was not statistically higher than that associated with the clinician-observed actions. The dotted line is the reference line (ie, no difference in weighted importance sampling between the algorithm rule and the clinician-observed actions). eICU-CRD indicates eICU Collaborative Research Database; MIMIC-IV, Medical Information Mart for Intensive Care; SOFA, Sequential Organ Failure Assessment; UCSF, University of California, San Francisco; UPMC, University of Pittsburgh Medical Center.

Figure 4.. Risk-Adjusted Odds of In-Hospital Mortality Comparing Concordance With the Reinforcement Learning Rule or a Simple Clinical Rule With Clinician-Observed Actions

Distribution using a regular standard error estimator or a robust standard error estimator. Reinforcement learning rule results displayed for combined validation cohort as well as each individual cohort. Simple clinical decision rule results displayed for combined validation cohort only. The risk-adjusted odds for in-hospital mortality were derived from inverse probability of treatment weighted pooled logistic regression models, adjusting for baseline and time-varying confounders. The results for vasopressin initiated per the reinforcement learning rule show the ORs for in-hospital mortality of concordance with the reinforcement learning rule in each 1-hour epoch compared with the clinician-observed actions for the overall external validation set and for each external validation dataset separately. The results for vasopressin initiated per the simple clinical rule show the ORs for in-hospital mortality of concordance with 3 independent simple clinical rules for vasopressin initiation in each 1-hour epoch for the overall external validation set. The 3 simple rules are: “initiate vasopressin when serum lactate is >4 mmol/L,” “initiate vasopressin when norepinephrine dose is >0.7 μg/kg/min,” and “initiate vasopressin when MAP is <65 mm Hg and time from shock onset is at least 12 hours.” eICU-CRD indicates eICU Collaborative Research Database; MAP, mean arterial pressure; MIMIC-IV, Medical Information Mart for Intensive Care; OR, odds ratio; UPMC, University of Pittsburgh Medical Center.