Validation of an Electronic Health Record-Based Machine Learning Model Compared With Clinical Risk Scores for Gastrointestinal Bleeding

Abstract

Background & aims: Guidelines recommend use of risk stratification scores for patients presenting with gastrointestinal bleeding (GIB) to identify very-low-risk patients eligible for discharge from emergency departments. Machine learning models may outperform existing scores and can be integrated within the electronic health record (EHR) to provide real-time risk assessment without manual data entry. We present the first EHR-based machine learning model for GIB.

Methods: The training cohort comprised 2546 patients and internal validation of 850 patients presenting with overt GIB (ie, hematemesis, melena, and hematochezia) to emergency departments of 2 hospitals from 2014 to 2019. External validation was performed on 926 patients presenting to a different hospital with the same EHR from 2014 to 2019. The primary outcome was a composite of red blood cell transfusion, hemostatic intervention (ie, endoscopic, interventional radiologic, or surgical), and 30-day all-cause mortality. We used structured data fields in the EHR, available within 4 hours of presentation, and compared the performance of machine learning models with current guideline-recommended risk scores, Glasgow-Blatchford Score, and Oakland Score. Primary analysis was area under the receiver operating characteristic curve. Secondary analysis was specificity at 99% sensitivity to assess the proportion of patients correctly identified as very low risk.

Results: The machine learning model outperformed the Glasgow-Blatchford Score (area under the receiver operating characteristic curve, 0.92 vs 0.89; P < .001) and Oakland Score (area under the receiver operating characteristic curve, 0.92 vs 0.89; P < .001). At the very-low-risk threshold of 99% sensitivity, the machine learning model identified more very-low-risk patients: 37.9% vs 18.5% for Glasgow-Blatchford Score and 11.7% for Oakland Score (P < .001 for both comparisons).

Conclusions: An EHR-based machine learning model performs better than currently recommended clinical risk scores and identifies more very-low-risk patients eligible for discharge from the emergency department.

Keywords: Electronic Health Record; Gastrointestinal Hemorrhage; Machine Learning.

Figure A4.1:

Calibration of conditional probabilities predicted by the machine learning model.

Figure A4.2:

Calibration of conditional probabilities predicted by the Glasgow-Blatchford Score.

Figure A4.3:

Calibration of conditional probabilities predicted by the Oakland score.

Figure 9.1:

Superimposed histograms displaying data distributions of 9 variables with differences that were significant by P<0.05 between the false positives and true negatives across the base variable characteristics reported above.

Figure 1:

Artificial Intelligence-Clinical Decision Support System – an Interactive Interpretable Machine Learning Dashboard. The inputs on the left show demographics, vital signs, and other input categories such as laboratory values, medications, and past medical history. The Top pane displays a patient that is not very low risk (probability is above the pre-set threshold for the high sensitivity 99%) and Bottom pane a very-low-risk patient. This is defined by the model probability output less than 6.3%, which is the output that corresponds to the 99% sensitivity threshold for very low risk patients. The top three variables that affect risk are displayed in graphical form, with how the risk changes across the different thresholds of the variable (e.g. as hemoglobin increases, risk of hospital-based intervention decreases)

Figure 2:

Flow diagram for the selected machine learning model (top left), deep learning model (top right), GBS (bottom left), and Oakland score (bottom right) with the respective predicted classifications, false positive, false negative, true positive, and true negative rates.