Explainable machine learning in outcome prediction of high-grade aneurysmal subarachnoid hemorrhage

Abstract

Objective: Accurate prognostic prediction in patients with high-grade aneruysmal subarachnoid hemorrhage (aSAH) is essential for personalized treatment. In this study, we developed an interpretable prognostic machine learning model for high-grade aSAH patients using SHapley Additive exPlanations (SHAP).

Methods: A prospective registry cohort of high-grade aSAH patients was collected in one single-center hospital. The endpoint in our study is a 12-month follow-up outcome. The dataset was divided into training and validation sets in a 7:3 ratio. Machine learning algorithms, including Logistic regression model (LR), support vector machine (SVM), random forest (RF), and extreme gradient boosting (XGBoost), were employed to develop a prognostic prediction model for high-grade aSAH. The optimal model was selected for SHAP analysis.

Results: Among the 421 patients, 204 (48.5%) exhibited poor prognosis. The RF model demonstrated superior performance compared to LR (AUC = 0.850, 95% CI: 0.783-0.918), SVM (AUC = 0.862, 95% CI: 0.799-0.926), and XGBoost (AUC = 0.850, 95% CI: 0.783-0.917) with an AUC of 0.867 (95% CI: 0.806-0 .929). Primary prognostic features identified through SHAP analysis included higher World Federation of Neurosurgical Societies (WFNS) grade, higher modified Fisher score (mFS) and advanced age, were found to be associated with 12-month unfavorable outcome, while the treatment of coiling embolization for aSAH drove the prediction towards favorable prognosis. Additionally, the SHAP force plot visualized individual prognosis predictions.

Conclusions: This study demonstrated the potential of machine learning techniques in prognostic prediction for high-grade aSAH patients. The features identified through SHAP analysis enhance model interpretability and provide guidance for clinical decision-making.

Keywords: SHapley additive exPlanations; aneurysmal subarachnoid hemorrhage; explainable machine learning; high-grade; prognosis prediction.

Figure 1

The flowchart of this study.

Figure 2

ROC curves for four machine learning models. (A) AUCs of four machine learning models in the training cohort; (B) AUCs of four machine learning models in the test cohort. ROC, receiver operating characteristic curve; AUC, area under the curve; LR, logistic regression; RF, random forest; SVM, support vector machine; XGB, extreme gradient boosting.

Figure 3

Decision curve analysis of random forest model. The black line is the net benefit for a strategy of treating all men; the yellow line is the net benefit of treating none. The y-axis indicates the overall net benefit, which is calculated by summing the benefits (true positive results and subtracting the harms (false positive results).

Figure 4

Summary plots of SHapley Additive exPlanations (SHAP) values. (A) SHAP feature importance quantified through the average absolute Shapley values. This plot illustrates the significance of each feature in development of the predictive model. (B) Representation of the influence exerted by each feature on the final model output, assessed via SHAP values distribution. Every individual patient is denoted by a data point within each row. The color indicates whether the continuous feature is at a high level (displayed in blue) or a low level (displayed in red) for that specific observation. When it comes to categorical features, the color blue signifies “yes”, while the color red corresponds to “no”. Location 1, 2, 3, 4, 5, 6, 7 denotes anterior cerebral artery, middle cerebral artery, internal cerebral artery, posterior cerebral artery, anterior communicating artery, posterior communicating artery and others, respectively.

Figure 5

SAHP dependency plot illustrating the top 5 clinical features in the random forest model. (A) WFNS 5; (B) Age; (C) mFS 4; (D) WFNS 2; (E) Treatment coiling. WFNS, World Federation of Neurosurgical Societies; mFS, modified Fisher scale.

Figure 6

SHAP force plot for interpreting individual’s prediction outcomes. This plot offers a visual illustration of the RF model’s predictions, wherein the red and blue bars signify risk factors and protective factors, respectively. The length of the bars corresponds to the extent of feature importance. (A) Poor outcome; (B) favorable outcome.