Do false positive alerts in naïve clinical decision support system lead to false adoption by physicians? A randomized controlled trial

Abstract

Objectives: False positive alerts in patient-safety-related clinical decision support systems (CDSS) are defined as alerts which incorrectly prompt when no-risk patients are encountered. It is an unfavorable condition which may potentially mislead physicians. The aim is to investigate physician responses toward false positive (FP) and true positive (TP) alerts in CDSS for the prevention of contrast-induced nephropathy (CIN).

Methods: A two-arm cluster randomized controlled trial was conducted in university hospitals. Eligible physicians were randomized to receive alert intervention or no intervention (groups 1 and 2, respectively). The alert system was embedded with a deliberately non-specific risk detection tool in order to generate TP and FP alerts. The naïve alert system would alert the physician to cancel the order regardless of the patient being at-risk or not at-risk. CIN risk was stratified as at-risk and no-risk according to a patient's pre-existing renal function. Contrast imaging order-cancellation rate was measured as primary outcome.

Results: 3802 contrast-enhanced examination orders from 66 physicians were analyzed. Demographic data and risk distributions of patients were similar and well-balanced between two groups. In the intervention group, a total of 1892 alerts were generated (332 TP alerts and 1560 FP alerts). Order-cancellation rates were 5.1% versus 1.4% in groups 1 and 2 for at-risk patients (relative risk [RR] = 3.69) from TP alerts, and 1.0% versus 1.4% for no-risk patients (RR = 0.71) from FP alerts. Using generalized linear model with generalized estimating equation, the FP alerts had no order-cancellation effect when compared to the control arm (adjusted RR = 0.69; 95%CI, 0.36-1.32). The TP alerts had a larger order-cancellation effect than that of the control arm (adjusted RR = 2.95; 95%CI, 0.94-9.27), which revealed a marginal trend toward significance. However, the effect was not statistically significant (adjusted RR = 1.24; 95%CI, 0.71-2.18) if TP and FP alerts were mixed.

Conclusions: Physicians are not likely to adopt recommendations provided by false positive alerts in patient-safety-related CDSS. If reporting only the adoption rate of CDSS as a whole without differentiating between TP and FP alerts, the effects of TP and FP alerts will be mixed, and thus, will lead to an underestimation of system effectiveness.

Keywords: Alert system; Clinical decision support system; Contrast-induced nephropathy; False positive alert; Patient safety; Randomized controlled trial.