Evaluating the effectiveness of clinical alerts: a signal detection approach

Abstract

Clinical alerts are widely used in healthcare to notify caregivers of critical information. Alerts can be presented through many different modalities, including verbal, paper and electronic. Increasingly, information technology is being used to automate alerts. Most applications, however, fall short in achieving the desired outcome. The objective of this study is twofold. First, we examine the effectiveness of verbal and written alerts in promoting adherence to infection control precautions during inpatient transfers to radiology. Second, we propose a quantitative framework based on Signal Detection Theory (SDT) for evaluating the effectiveness of clinical alerts. Our analysis shows that verbal alerts are much more effective than written alerts. Further, using precaution alerts as a case study, we demonstrate the application of SDT to evaluate the quality of alerts, and human behavior in handling alerts. We hypothesize that such technique can improve our understanding of computerized alert systems, and guide system redesign.

Figure 1.

Graphical representation of Signal Detection Theory.

Figure 2.

2 x 2 contingency table, representing the 4 possible scenarios (S=alert is provided, ∼S=alert is not provided, R=alert is adhered to, ∼R=alert is not adhered to).