EASY-APP: An artificial intelligence model and application for early and easy prediction of severity in acute pancreatitis

Abstract

Background: Acute pancreatitis (AP) is a potentially severe or even fatal inflammation of the pancreas. Early identification of patients at high risk for developing a severe course of the disease is crucial for preventing organ failure and death. Most of the former predictive scores require many parameters or at least 24 h to predict the severity; therefore, the early therapeutic window is often missed.

Methods: The early achievable severity index (EASY) is a multicentre, multinational, prospective and observational study (ISRCTN10525246). The predictions were made using machine learning models. We used the scikit-learn, xgboost and catboost Python packages for modelling. We evaluated our models using fourfold cross-validation, and the receiver operating characteristic (ROC) curve, the area under the ROC curve (AUC), and accuracy metrics were calculated on the union of the test sets of the cross-validation. The most critical factors and their contribution to the prediction were identified using a modern tool of explainable artificial intelligence called SHapley Additive exPlanations (SHAP).

Results: The prediction model was based on an international cohort of 1184 patients and a validation cohort of 3543 patients. The best performing model was an XGBoost classifier with an average AUC score of 0.81 ± 0.033 and an accuracy of 89.1%, and the model improved with experience. The six most influential features were the respiratory rate, body temperature, abdominal muscular reflex, gender, age and glucose level. Using the XGBoost machine learning algorithm for prediction, the SHAP values for the explanation and the bootstrapping method to estimate confidence, we developed a free and easy-to-use web application in the Streamlit Python-based framework (http://easy-app.org/).

Conclusions: The EASY prediction score is a practical tool for identifying patients at high risk for severe AP within hours of hospital admission. The web application is available for clinicians and contributes to the improvement of the model.

Keywords: acute pancreatitis; artificial intelligence; severity prediction.

FIGURE 1

The workflow of the development of the prediction model

FIGURE 2

The cross‐validated (fold = 4) receiver operating characteristic (ROC) curve of the XGBoost model. The corresponding mean area under the curve (AUC) is 0.809. The 95% confidence interval is [0.776, 0.842]

FIGURE 3

The performance of the XGBoost model trained on different sized sets. The points show the area under the curve (AUC) scores, and the bars are the corresponding confidence intervals

FIGURE 4

The performance of the model using varying numbers of attributes with the top k most important features. The importance is calculated using the SHapley Additive exPlanations (SHAP) importance. The points show the area under the curve (AUC) scores, and the bars are the corresponding confidence intervals

FIGURE 5

The predicted severity score on a selected subset of the dataset and the confidence intervals for the 10th and 90th percentiles and the 25th and 75th percentiles. The records are sorted by the severity score

FIGURE 6

A summary plot of the impact of the features on the prediction (severity score) of the model. Each patient is represented by a point in each row. The colour of the points represents the relative value of the feature, and the x‐position of the points is the SHapley Additive exPlanations (SHAP) value, that is, the impact on the model's prediction

FIGURE 7

Three examples of the local explanation of the predictions using the SHapley Additive exPlanations (SHAP) values. (A) Predicted mild acute pancreatitis (AP). (B) Predicted AP with borderline severity. (C) Predicted severe AP. Factors that push the predicted score higher compared to the base value (mean prediction) are coloured orange, and those pushing lower the prediction are shown in green

FIGURE 8

An example output of the web application for the following input parameters—age: 55 years, gender: 0 (woman), body mass index: 22, alcohol consumption: 1 (true), blood pressure/pulse: 140/75/60, body temperature: 37.0°C, respiratory rate: 25. (A) Predicted severity score. (B) Explanation of prediction. (C) The kernel density estimate plot of the distribution of the predictions