Effects of a computerized feedback intervention on safety performance by junior doctors: results from a randomized mixed method study

Abstract

Background: The behaviour of doctors and their responses to warnings can inform the effective design of Clinical Decision Support Systems. We used data from a University hospital electronic prescribing and laboratory reporting system with hierarchical warnings and alerts to explore junior doctors' behaviour. The objective of this trial was to establish whether a Junior Doctor Dashboard providing feedback on prescription warning information and laboratory alerting acceptance rates was effective in changing junior doctors' behaviour.

Methods: A mixed methods approach was employed which included a parallel group randomised controlled trial, and individual and focus group interviews. Junior doctors below the specialty trainee level 3 grade were recruited and randomised to two groups. Every doctor (N = 42) in the intervention group was e-mailed a link to a personal dashboard every week for 4 months. Nineteen participated in interviews. The 44 control doctors did not receive any automated feedback. The outcome measures were the difference in responses to prescribing warnings (of two severities) and laboratory alerting (of two severities) between the months before and the months during the intervention, analysed as the difference in performance between the intervention and the control groups.

Results: No significant differences were observed in the rates of generating prescription warnings, or in the acceptance of laboratory alarms. However, responses to laboratory alerts differed between the pre-intervention and intervention periods. For the doctors of Foundation Year 1 grade, this improvement was significantly (p = 0.002) greater in the group with access to the dashboard (53.6% ignored pre-intervention compared to 29.2% post intervention) than in the control group (47.9% ignored pre-intervention compared to 47.0% post intervention). Qualitative interview data indicated that while junior doctors were positive about the electronic prescribing functions, they were discriminating in the way they responded to other alerts and warnings given that from their perspective these were not always immediately clinically relevant or within the scope of their responsibility.

Conclusions: We have only been able to provide weak evidence that a clinical dashboard providing individualized feedback data has the potential to improve safety behaviour and only in one of several domains. The construction of metrics used in clinical dashboards must take account of actual work processes.

Trial registration isrctn: ISRCTN72253051.

Figure 1

Example of a password warning graph. The prescription warning information displays 6 dials for disallowed and password warnings for each category of allergy/contraindications (combined), excessive dosing, and drug interactions.

Figure 2

Example of a laboratory alarm dial. The laboratory alerting section contains 2 dials with the percentage of ignored alarms and alerts.

Figure 3

Flow of participants through trial. This figure adheres to the consort statement and shows the phases of recruitment for the trial.

Figure 4

Results of alerts and alarms ignored by junior doctors. Plot of the proportion of laboratory alerts and alarms ignored by the junior doctors during the trial. The quoted p-values are those relating to the interaction terms in the generalised estimating equations.