Identification of Patients Expected to Benefit from Electronic Alerts for Acute Kidney Injury

Abstract

Background and objectives: Electronic alerts for heterogenous conditions such as AKI may not provide benefit for all eligible patients and can lead to alert fatigue, suggesting that personalized alert targeting may be useful. Uplift-based alert targeting may be superior to purely prognostic-targeting of interventions because uplift models assess marginal treatment effect rather than likelihood of outcome.

Design, setting, participants, & measurements: This is a secondary analysis of a clinical trial of 2278 adult patients with AKI randomized to an automated, electronic alert system versus usual care. We used three uplift algorithms and one purely prognostic algorithm, trained in 70% of the data, and evaluated the effect of targeting alerts to patients with higher scores in the held-out 30% of the data. The performance of the targeting strategy was assessed as the interaction between the model prediction of likelihood to benefit from alerts and randomization status. The outcome of interest was maximum relative change in creatinine from the time of randomization to 3 days after randomization.

Results: The three uplift score algorithms all gave rise to a significant interaction term, suggesting that a strategy of targeting individuals with higher uplift scores would lead to a beneficial effect of AKI alerting, in contrast to the null effect seen in the overall study. The prognostic model did not successfully stratify patients with regards to benefit of the intervention. Among individuals in the high uplift group, alerting was associated with a median reduction in change in creatinine of -5.3% (P=0.03). In the low uplift group, alerting was associated with a median increase in change in creatinine of +5.3% (P=0.005). Older individuals, women, and those with a lower randomization creatinine were more likely to receive high uplift scores, suggesting that alerts may benefit those with more slowly developing AKI.

Conclusions: Uplift modeling, which accounts for treatment effect, can successfully target electronic alerts for AKI to those most likely to benefit, whereas purely prognostic targeting cannot.

Keywords: Acute Kidney Injury; Adult; Alert; Algorithms; Clinical Decision Support; Female; Humans; Personalized Medicine; Precision Medicine; Probability; Prognosis; Random Allocation; creatinine; outcomes.

Graphical abstract

Figure 1.

As targeting becomes more stringent, the effect on change in creatinine between alert and control becomes more pronounced in the uplift models, but remains stable in the purely prognostic model. Comparisons of change in creatinine in usual care and alert groups as the targeted population narrows. Effect of targeting all patients with AKI is the center of the x-axis (100%). To the left of center, the effect of targeting those with lower uplift scores is visualized. To the right of center, the effect of targeting those with higher uplift scores is visualized. Where the red line is lower than the blue line, a benefit of alerting is present, and vice versa. Charts are truncated at 20% given sparse data beyond those points. (A) T-Learner (P for interaction =0.02); (B) Z-Learner (P for interaction =0.01); (C) X-Learner (P for interaction =0.03); (D) Prognostic targeting (P for interaction =0.30).

Figure 2.

Alerting results in less of change in creatinine than usual care in the group identified as likely to benefit, while the reverse is true in the group identified as unlikely to benefit. Change in creatinine from randomization to 3 days later, stratified by alert status in the group predicted unlikely to benefit from alerts versus likely to benefit from alerts on the basis of the Z-learner uplift algorithm. Bars represent median and whiskers represent IQR.