MySurgeryRisk: Development and Validation of a Machine-learning Risk Algorithm for Major Complications and Death After Surgery

Abstract

Objective: To accurately calculate the risk for postoperative complications and death after surgery in the preoperative period using machine-learning modeling of clinical data.

Background: Postoperative complications cause a 2-fold increase in the 30-day mortality and cost, and are associated with long-term consequences. The ability to precisely forecast the risk for major complications before surgery is limited.

Methods: In a single-center cohort of 51,457 surgical patients undergoing major inpatient surgery, we have developed and validated an automated analytics framework for a preoperative risk algorithm (MySurgeryRisk) that uses existing clinical data in electronic health records to forecast patient-level probabilistic risk scores for 8 major postoperative complications (acute kidney injury, sepsis, venous thromboembolism, intensive care unit admission >48 hours, mechanical ventilation >48 hours, wound, neurologic, and cardiovascular complications) and death up to 24 months after surgery. We used the area under the receiver characteristic curve (AUC) and predictiveness curves to evaluate model performance.

Results: MySurgeryRisk calculates probabilistic risk scores for 8 postoperative complications with AUC values ranging between 0.82 and 0.94 [99% confidence intervals (CIs) 0.81-0.94]. The model predicts the risk for death at 1, 3, 6, 12, and 24 months with AUC values ranging between 0.77 and 0.83 (99% CI 0.76-0.85).

Conclusions: We constructed an automated predictive analytics framework for machine-learning algorithm with high discriminatory ability for assessing the risk of surgical complications and death using readily available preoperative electronic health records data. The feasibility of this novel algorithm implemented in real time clinical workflow requires further testing.

Figure 1

Figure 1A. The conceptual framework of MySurgeryRisk analytics platform. The diagram shows sequence of steps from aggregation of raw data, data engineering and data analytics to final output. Figure 1B. The conceptual diagram of the Intelligent Perioperative Platform. This platform resides in a secure environment and in real time integrates and transforms electronic health records data, runs predictive algorithms, produces outputs for physicians, inputs their feedback and prospectively collects data for the future retraining of the prediction models

Figure 1

Figure 1A. The conceptual framework of MySurgeryRisk analytics platform. The diagram shows sequence of steps from aggregation of raw data, data engineering and data analytics to final output. Figure 1B. The conceptual diagram of the Intelligent Perioperative Platform. This platform resides in a secure environment and in real time integrates and transforms electronic health records data, runs predictive algorithms, produces outputs for physicians, inputs their feedback and prospectively collects data for the future retraining of the prediction models

Figure 2

MySurgeryRisk Output. The sample ouput for subjects with A, low mortality risk, and B, high mortality risk. Figure shows the predicted risks for eight postoperative complications for the given patient in eight equal-sized pies. The calculated cutoff values for AKI, ICU, MV, WND, CV, NEU, SEP, and VTE, were 0.35, 0.35, 0.13, 0.1, 0.07, 0.07, 0.06, and 0.03 respectively. Subjects are classified as high risk for a complication if calculated risk score exceeds the respective cutoff and respective pie is marked as red, and green otherwise. The size of the pie represents the proportion of the risk, scaled based on the cutoff for each complication. Green background color represents low mortality risk (Figure 2A) whereas red background color shows high mortality risk (Figure 2B). Abbreviation: AKI, acute kidney injury, CV, cardiovascular complications, ICU, intensive care unit addmission > 48 hours, MV, mechanical ventilation > 48 hours, NEU, neurologic complications, SEP, sepsis, VTE, venous thromboembolism, WND, wound complications.

Figure 3

Receiver operating characteristic curves and performance metrics for MySurgeryRisk algorithm in predicting A, more prevalent complications, B, less prevalent complications, and C, mortality.

Figure 3

Receiver operating characteristic curves and performance metrics for MySurgeryRisk algorithm in predicting A, more prevalent complications, B, less prevalent complications, and C, mortality.

Figure 3

Receiver operating characteristic curves and performance metrics for MySurgeryRisk algorithm in predicting A, more prevalent complications, B, less prevalent complications, and C, mortality.