External Validation of Persistent Severe Acute Kidney Injury Prediction With Machine Learning Model

Abstract

Objective: To externally validate the persistent electronic alert (PersEA) machine learning model for predicting persistent severe acute kidney injury (psAKI), addressing the scarcity of validated prediction models.

Patients and methods: We included adult patients (18 years or older) admitted to intensive care unit with at least stage 2 acute kidney injury (AKI) at a tertiary medical center, using retrospective data collected between January 1st, 2017 and December 31st, 2022. The data were accessed and analyzed during the period from March 1st, 2023, through July 28th, 2023. The psAKI was defined as AKI stage 3 lasting for ≥72 hours or AKI leading to death in 48 hours or kidney replacement therapy in 1 day. The performance of the PersEA model, a boosted tree algorithm fed by hourly patient data via electronic health records to provide real-time psAKI predictions, was evaluated using specific metrics that penalize late alarms. We measured the area under the receiver operating characteristic and the area under the precision-recall curves.

Results: After screening, 4479 patients from the Mayo Clinic cohort were included in the current external validation study, with 234 (5.22%) having psAKI. Results from the Amsterdam UMCdb (531 patients, 59 [11.11%] positive) and MIMIC-III (495 patients, 57 [11.52%] positive) cohorts were obtained in a prior development study. The model demonstrated an area under the receiver operating characteristic curve of 0.98 (95% CI, 0.97-0.98) and an area under the precision-recall curve of 0.67 (95% CI, 0.60-0.73), and when applying the threshold that reached 0.80 sensitivity on the internal cohort, PersEA achieved 0.88 sensitivity, 0.94 specificity, and 0.47 precision, all based on Mayo Clinic data.

Conclusion: The PersEA model performed excellently on an external cohort, showing that it is scalable on high-quality data with little to no tuning once a noisy training set is chosen.

Figure 1

Risk rate analysis for a positive patient with time of persistent event (T_p) at the 47th intensive care unit hour. The medium threshold triggers an alert at the 44th hour (−3 hours lead time), whereas the low threshold does so at the 39th hour (−8 hours lead time). However, the high threshold prompts an alert at the 65th hour (18 hours lead time), potentially too late clinically. Notably, the low and medium thresholds result in true-positive results, whereas the high threshold leads to a false-negative result according to our alarm definition.

Figure 2

Performance curves of persistent electronic alert on Mayo Clinic for each baseline serum creatinine definition. Dots represent 80% of the sensitivity threshold working point. ADM, creatinine at admission used as the baseline; FPR, false positive rate; MDRD, modification of diet in renal disease; OPAL, Optimization Presumption Adjustment Limitation Consideration (Definition from the OPAL study [NCT01846884]); PR, precision-recall curve; ROC, receiver operating characteristic; UCM, U-Care Medical definition.