Toward Dynamic Risk Prediction of Outcomes After Coronary Artery Bypass Graft: Improving Risk Prediction With Intraoperative Events Using Gradient Boosting

Abstract

Background: Intraoperative data may improve models predicting postoperative events. We evaluated the effect of incorporating intraoperative variables to the existing preoperative model on the predictive performance of the model for coronary artery bypass graft.

Methods: We analyzed 378 572 isolated coronary artery bypass graft cases performed across 1083 centers, using the national Society of Thoracic Surgeons Adult Cardiac Surgery Database between 2014 and 2016. Outcomes were operative mortality, 5 postoperative complications, and composite representation of all events. We fitted models by logistic regression or extreme gradient boosting (XGBoost). For each modeling approach, we used preoperative only, intraoperative only, or pre+intraoperative variables. We developed 84 models with unique combinations of the 3 variable sets, 2 variable selection methods, 2 modeling approaches, and 7 outcomes. Each model was tested in 20 iterations of 70:30 stratified random splitting into development/testing samples. Model performances were evaluated on the testing dataset using the C statistic, area under the precision-recall curve, and calibration metrics, including the Brier score.

Results: The mean patient age was 65.3 years, and 24.7% were women. Operative mortality, excluding intraoperative death, occurred in 1.9%. In all outcomes, models that considered pre+intraoperative variables demonstrated significantly improved Brier score and area under the precision-recall curve compared with models considering pre or intraoperative variables alone. XGBoost without external variable selection had the best C statistics, Brier score, and area under the precision-recall curve values in 4 of the 7 outcomes (mortality, renal failure, prolonged ventilation, and composite) compared with logistic regression models with or without variable selection. Based on the calibration plots, risk restratification for mortality showed that the logistic regression model underestimated the risk in 11 114 patients (9.8%) and overestimated in 12 005 patients (10.6%). In contrast, the XGBoost model underestimated the risk in 7218 patients (6.4%) and overestimated in 0 patients (0%).

Conclusions: In isolated coronary artery bypass graft, adding intraoperative variables to preoperative variables resulted in improved predictions of all 7 outcomes. Risk models based on XGBoost may provide a better prediction of adverse events to guide clinical care.

Keywords: benchmarking; coronary artery bypass; machine learning; medical informatics; probability learning.

Figure 1:

Analysis flow for development and evaluation of models. The figure summarizes the modeling approach and metrics used to evaluate the performance. Combinations of variable sets, variable selection approach, and modeling technique for 7 outcomes resulted in 84 different models. CABG = coronary artery bypasss graft surgery; STS ACSD= Society of Thoracic Surgeons Adult Cardiac Surgery Database; AUPRC = area under the precision-recall curve

Figure 2:

Model performances for mortality and renal failure. Figure summarizes model performances for mortality and renal failure evaluated by 3 metrics. Circled red triangles represent the baseline model, which are logistic regression models using preoperative variables only without further variable selection. * indicates the model with best performance within the same variable set. For all metrics, the right end of the x-axis is better and the left end is worse. For example, for operative mortality, XGBoost model using pre+intraoperative variables without variable selection had the best performance in c-statistics, Brier score, and the area under precision-recall curve (AUPRC).

Figure 3:

Continuous calibration plot for 7 outcomes. Figure shows continuous calibration plot for mortality (left) and renal failure (right). Red lines are XGBoost and blue lines are logistic regression model calibrations. Dotted lines are models using preoperative variables only and solid lines are models using pre+intraoperative variables. Black line represents perfect calibration. The legend shows percent of the cohort that had predicted event probability above the indicate threshold in percentage. For example, for operative mortality, 2.3% of the patients had predicted probability of operative mortality >10%.

Figure 4:

Shift table of predicted risk for operative mortality: Logistic regression with preoperative variables vs. XGBoost with pre+intraoperative variables. The figure shows predicted risk of operative mortality by the base model (logistic regression using preoperative variables without variable selection) and the best model (XGBoost using pre+intraoperative variables without variable selection). Actual observed mortality rate is indicated by the % and numbers in parenthesis indicate number of all patients in each predicted risk strata. Gray cells are those classified in the same stratum by both models. Base model underestimated 11,114 patients (9.8%) and overestimated 12,005 patients (10.6%). Best model underestimated 7,218 patients (6.4%) and overestimated 0 patients (0%). *1–5 denotes cell location by the column and a-e denotes cell location by the row.

Figure 5:

Shift table of predicted risk for operative mortality: Logistic regression with pre+intraoperative variables vs. XGBoost with pre+intraoperative variables. The figure shows predicted risk of operative mortality by the base model (logistic regression using preoperative variables without variable selection) and the best model (XGBoost using pre+intraoperative variables without variable selection). Actual observed mortality rate is indicated by the % and numbers in parenthesis indicate number of all patients in each predicted risk strata. Gray cells are those classified in the same stratum by both models. Base model underestimated 7,137 patients (6.3%) and overestimated 3,566 patients (3.1%). Best model underestimated 4,263 patients (3.8%) and overestimated 1,886 patients (1.7%). *1–5 denotes cell location by the column and a-e denotes cell location by the row.