Validation of a machine learning model for predicting gastrointestinal bleeding in patients with direct oral anticoagulants

Abstract

Background and aim: Direct oral anticoagulants (DOACs) carry a risk of gastrointestinal bleeding (GIB). We aimed to develop and validate machine learning (ML) models to predict GIB in DOAC users and compare them with conventional risk scores.

Methods: We retrospectively analyzed 4,494 patients aged ≥18 years prescribed DOACs from December 2014 to October 2020. Patients were allocated to the training (n = 3,147), internal (n = 677), and external (n = 670) validation cohorts. Three ML algorithms, Gradient Boosting Machine (GBM), XGBoost, and Generalized Linear Model (GLM), predicted GIB at 12 and 24 months. Performance was assessed using the area under the receiver operating characteristic curve (AUC) and specificity at 100% sensitivity, compared with the HAS-BLED, ATRIA, VTE-BLEED, and ORBIT scores.

Results: At 24 months, XGBoost achieved the AUCs in the training (0.862), internal validation (0.819), and external validation (0.905) sets. At 12 months, XGBoost performed with AUCs of 0.917, 0.839, and 0.948, respectively. XGBoost exceeded the conventional scores, although ORBIT was the best among the latter (AUC 0.780 at 24 months, 0.728 at 12 months). The ML models also achieved higher specificity at 100% sensitivity. At 12 months, XGBoost and GB model demonstrated 79.8% specificity at 100% sensitivity, whereas GLM showed 67.8%. The conventional models were lower, with an ORBIT of 39.8%. By 24 months, GLM and ORBIT specificities were 43.8% and 40.0%, respectively.

Conclusions: ML models, particularly XGBoost, outperformed traditional bleeding risk scores in predicting GIB in DOAC users. However, the performance of the ML models was unsatisfactory. Further research is warranted to achieve a better performance.

Keywords: Anticoagulant; gastrointestinal hemorrhage; machine learning; retrospective study; risk assessment.