Decision analysis and reinforcement learning in surgical decision-making

Abstract

Background: Surgical patients incur preventable harm from cognitive and judgment errors made under time constraints and uncertainty regarding patients' diagnoses and predicted response to treatment. Decision analysis and techniques of reinforcement learning theoretically can mitigate these challenges but are poorly understood and rarely used clinically. This review seeks to promote an understanding of decision analysis and reinforcement learning by describing their use in the context of surgical decision-making.

Methods: Cochrane, EMBASE, and PubMed databases were searched from their inception to June 2019. Included were 41 articles about cognitive and diagnostic errors, decision-making, decision analysis, and machine-learning. The articles were assimilated into relevant categories according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines.

Results: Requirements for time-consuming manual data entry and crude representations of individual patients and clinical context compromise many traditional decision-support tools. Decision analysis methods for calculating probability thresholds can inform population-based recommendations that jointly consider risks, benefits, costs, and patient values but lack precision for individual patient-centered decisions. Reinforcement learning, a machine-learning method that mimics human learning, can use a large set of patient-specific input data to identify actions yielding the greatest probability of achieving a goal. This methodology follows a sequence of events with uncertain conditions, offering potential advantages for personalized, patient-centered decision-making. Clinical application would require secure integration of multiple data sources and attention to ethical considerations regarding liability for errors and individual patient preferences.

Conclusion: Traditional decision-support tools are ill-equipped to accommodate time constraints and uncertainty regarding diagnoses and the predicted response to treatment, both of which often impair surgical decision-making. Decision analysis and reinforcement learning have the potential to play complementary roles in delivering high-value surgical care through sound judgment and optimal decision-making.

Figure 1:. Optimizing the accuracy of the prediction model may not optimize clinical utility.

Model A has greater accuracy, but if a pregnant woman presenting with fever and right-sided abdominal pain wishes to avoid fetal demise due to a wrong diagnosis, then Model B is favorable.

Figure 2:. Decision tree framework and clinical application.

When it is unclear whether a diagnostic or therapeutic intervention is useful, decision tree analysis identifies a probability threshold (pt) at which value-adjusted outcomes for intervention and no intervention are equivocal. A prediction model or published literature provides the probability that disease is present. If this value is greater than pt, then the intervention is useful. Published literature and patient interviews provide relative values for each outcome.

Figure 3:. Reinforcement learning framework and clinical application.

An algorithm interacts with its environment (consisting of data from electronic health records or datasets) to learn states (representing disease or patient acuity), actions that lead to new states, probabilities of transitioning between states, and associations between state transitions and an ultimate goal, such as survival or discharge to home in good health. The algorithm then identifies actions that are most likely to achieve the ultimate goal. This process can occur within a Markov Decision Process framework and apply to a patient presenting with bowel obstruction, estimating the clinical utility of observation and operative exploration in response to evolving clinical conditions.

Figure 4:. Comparison of decision analysis and reinforcement learning for augmenting clinical reasoning.

The unique strengths and weaknesses of decision analysis and reinforcement learning suggest complementary roles in augmenting clinical reasoning.

Figure 5:. Decision analysis and deep reinforcement learning have complementary roles in augmenting population-based and personalized decision-making.

Input variables from patient assessments and the data from the electronic health record feed decision analysis tools that calculate probability thresholds to inform population-based recommendations. Reinforcement learning models combined with deep learning representation of an expanded set of input data can identify actions yielding the greatest probability of a patient-centered outcome.