Comparing clinical judgment with the MySurgeryRisk algorithm for preoperative risk assessment: A pilot usability study

Abstract

Background: Major postoperative complications are associated with increased cost and mortality. The complexity of electronic health records overwhelms physicians' abilities to use the information for optimal and timely preoperative risk assessment. We hypothesized that data-driven, predictive-risk algorithms implemented in an intelligent decision-support platform simplify and augment physicians' risk assessments.

Methods: This prospective, nonrandomized pilot study of 20 physicians at a quaternary academic medical center compared the usability and accuracy of preoperative risk assessment between physicians and MySurgeryRisk, a validated, machine-learning algorithm, using a simulated workflow for the real-time, intelligent decision-support platform. We used area under the receiver operating characteristic curve to compare the accuracy of physicians' risk assessment for six postoperative complications before and after interaction with the algorithm for 150 clinical cases.

Results: The area under the receiver operating characteristic curve of the MySurgeryRisk algorithm ranged between 0.73 and 0.85 and was significantly better than physicians' initial risk assessments (area under the receiver operating characteristic curve between 0.47 and 0.69) for all postoperative complications except cardiovascular. After interaction with the algorithm, the physicians significantly improved their risk assessment for acute kidney injury and for an intensive care unit admission greater than 48 hours, resulting in a net improvement of reclassification of 12% and 16%, respectively. Physicians rated the algorithm as easy to use and useful.

Conclusion: Implementation of a validated, MySurgeryRisk computational algorithm for real-time predictive analytics with data derived from the electronic health records to augment physicians' decision-making is feasible and accepted by physicians. Early involvement of physicians as key stakeholders in both design and implementation of this technology will be crucial for its future success.

Figure 1A.

Design of Intelligent Perioperative Platform that hosts MySurgeryRisk Algorithm, reprinted with permission from Annals of Surgery, Bihorac et al.

Figure 1B.

The interactive interface for physicians to input their initial assessment of absolute risk for each complication.

Figure 1C.

The interactive interface displaying absolute risk scores and risk groups calculated by MySurgeryRisk algorithm to physicians. Each score was accompanied with the display of the top three features that were the most important contributors to calculated risk for the individual patient.

Figure 1D.

The interactive interface for physicians to input their repeated assessment of the absolute risk after reviewing MySurgeryRisk scores for the same case.

Figure 2.

Flowchart of the study design.