An artificial intelligence model to predict mortality among hemodialysis patients: A retrospective validated cohort study

Abstract

Hemodialysis stands as the most prevalent renal replacement therapy globally. Accurately identifying mortality among hemodialysis patients is paramount importance, as it enables the formulation of tailored interventions and facilitates timely management. The objective of the study was to establish and validate an artificial intelligence (AI) model to predict mortality among hemodialysis patients. The data of 559 patients with hemodialysis at a large tertiary hospital were retrospectively analyzed, and those of 82 patients were extracted from another tertiary hospital. The patients from the large tertiary hospital constituted the model development cohort, and the patients from another tertiary hospital constituted the external validation cohort. The patients in the model development cohort were randomly divided into a training cohort and an internal validation cohort at a ratio of 8:2. The machine learning algorithms used to develop the models for the training group included logistic regression (LR), decision tree (DT), extreme gradient boosting machine (eXGBM), neural network (NN), and support vector machine (SVM). The predictive performances of all the models were evaluated using discrimination and calibration. In addition, a comprehensive scoring system to evaluate the prediction performance of the model was also used, the scoring system had the scores ranging from 0 to 50. The optimal model had the highest total score for the internal and external validation, and was further deployed as an AI application using Streamlit. The rates of mortality at one year, four years, and seven years in the model development group were determined to be 2.68%, 15.38%, and 33.09%, respectively. The model, which predicted mortality at these time points, achieved impressive area under the curve (AUC) values of 0.979 (95% CI: 0.959-0.998), 0.933 (95% CI: 0.916-0.958), and 0.935 (95% CI: 0.895-0.976), respectively, using the eXGBM model. The corresponding accuracies were 0.931, 0.889, and 0.931, with precision values of 0.891, 0.857, and 0.891, and brier scores of 0.051, 0.096, and 0.051, respectively. Notably, the eXGBM model outperformed other models with a score of 46 in the comprehensive scoring system, followed by the NN model with a score of 35. External validation further confirmed the robust predictive performance of the eXGBM model, with an AUC value of 0.892 (95% CI: 0.840-0.945). The eXGBM model emerged as the most reliable predictor of mortality among hemodialysis patients in this study. This model has been made available online at https://mortality-among-hemodialysis-bpypyb4dxvq4hja29kwsev.streamlit.app/ . Users can simply access the link, input relevant features, and receive predictions on mortality risk. Furthermore, the AI model provides insights into how the predictions were generated and offers personalized recommendations for intervention strategies. This study has successfully developed and validated an AI application for assessing mortality risk in hemodialysis patients. This tool empowers healthcare professionals to promptly identify individuals at high risk of mortality, thereby aiding in clinical decision-making and intervention planning. For patients at high risk of early death, caution is advised when considering kidney transplant surgery. Conversely, for those with a high probability of extended survival, kidney transplant surgery may present a favorable treatment option.

Keywords: External validation; Hemodialysis; Machine learning; Mortality; Prediction models.

Fig. 1

The experimental design consisted of three main components: data collection, randomization, and modelling and validation. The study employed various techniques for modeling, including the logistic regression (LR) model, as well as four machine learning models: neural network (NN), decision tree (DT), extreme gradient boosting machine (eXGBM), and support vector machine (SVM).

Fig. 2

Area under the curve (AUC) achieved by different models after applying 100 bootstraps. The horizontal axis of the graph indicates the quantity of bootstraps, whereas the vertical axis indicates the AUC value. Every model is distinguished by a unique color on the graph.

Fig. 3

Heatmap of the scoring system for comprehensively evaluating the prediction performance of all models. The scoring system utilized 10 evaluation metrics, with each metric being assessed on a scale from 1 to 5, where higher ratings indicated better predictive capabilities. Within this visual representation, green is indicative of comparatively subpar prediction performance, whereas red signifies relatively strong prediction performance.

Fig. 4

Density curves for all the models. A. Logistic regression; B. eXtreme gradient boosting machine; C. Support vector machine; D. Neural network; E. Decision tree. Green color represents patients with a low risk of 4-year mortality, while red color represents patients with a high risk of 4-year mortality. The less overlap between the red and green colors signifies a higher discriminatory ability of the model.

Fig. 5

Calibration curves and histograms of mean prediction probabilities for all the models. A. Calibration curve; B. Logistic regression; C. eXtreme gradient boosting machine; D. Support vector machine; E. Neural network; F. Decision tree. The calibration curve in this context illustrates the relationship between the average predicted probability and the fraction of positive outcomes (such as events or occurrences) within each group of ten individuals (deciles). This curve is particularly useful for assessing how well the predicted probabilities align with the actual probabilities of positive outcomes. On the other hand, the histogram accompanying the calibration curve displays the distribution of mean predicted probabilities across different bins or intervals, along with the count of observations falling within each interval.

Fig. 6

Decision curve analysis for all the models in the internal validation cohort. In the decision curve analysis, the dotted black line represents the “treat none” scenario, which indicates the net benefit when no individuals receive treatment. This line serves as a reference point for comparing the benefit of different strategies. On the other hand, the solid black line represents the “treat all” scenario, where all individuals are treated regardless of their predicted risk. This line helps in understanding the net benefit of applying the model to all individuals and treating them based on the predicted probabilities. By comparing the solid and dotted black lines with the curve generated by the models, decision-makers can assess the clinical utility and net benefit of using the predictive model to guide treatment decisions at different threshold probabilities.

Fig. 7

Feature importance analysis using the SHapley Additive exPlanation (SHAP). A. Model development cohort; B. Internal validation cohort. The X-axis of the graph depicts the SHAP value, while the Y-axis represents the model predictors. Green signifies a low feature value, whereas red denotes a high feature value. IHD stands for Ischemic Heart Disease, EF represents Ejection Fraction, and N denotes Neutrophil.

Fig. 8

The online application is deployed with the optimal machine learning model. It consists of four main interfaces: a panel for selecting model parameters (A), an interface for calculating the probability (B), an interface for showing model explanation, and an interface for introducing the model (C). To use the application, users can choose the model parameters they are interested in through the parameter selection panel. After setting the parameters, users can click the “Submit” button to start the calculation process. The application will then calculate and display the anticipated probability of one-, four-, and seven-year mortality in hemodialysis patients.