Formative Usability Testing Reduces Severe Blood Product Ordering Errors

Abstract

Background: Medical errors in blood product orders and administration are common, especially for pediatric patients. A failure modes and effects analysis in our health care system indicated high risk from the electronic blood ordering process.

Objectives: There are two objectives of this study as follows:(1) To describe differences in the design of the original blood product orders and order sets in the system (original design), new orders and order sets designed by expert committee (DEC), and a third-version developed through user-centered design (UCD).(2) To compare the number and type of ordering errors, task completion rates, time on task, and user preferences between the original design and that developed via UCD.

Methods: A multidisciplinary expert committee proposed adjustments to existing blood product order sets resulting in the DEC order set. When that order set was tested with front-line users, persistent failure modes were detected, so orders and order sets were redesigned again via formative usability testing. Front-line users in their native clinical workspaces were observed ordering blood in realistic simulated scenarios using a think-aloud protocol. Iterative adjustments were made between participants. In summative testing, participants were randomized to use the original design or UCD for five simulated scenarios. We evaluated differences in ordering errors, time on task, and users' design preference with two-sample t-tests.

Results: Formative usability testing with 27 providers from seven specialties led to 18 changes made to the DEC to produce the UCD. In summative testing, error-free task completion for the original design was 36%, which increased to 66% in UCD (30%, 95% confidence interval [CI]: 3.9-57%; p = 0.03). Time on task did not vary significantly.

Conclusion: UCD led to substantially different blood product orders and order sets than DEC. Users made fewer errors when ordering blood products for pediatric patients in simulated scenarios when using the UCD orders and order sets compared with the original design.

Fig. 1

Progression of order and order set design. We began with the original blood product orders and order sets, in which blood product orders were distributed across three main order sets (see Supplementary Figs. S1 A–C and S2 A and B , available in the online version). In design by expert committee (DEC), we condensed these into a single order set and made minor adjustments in the “prepare” and “transfuse” orders (see Supplementary Fig. S1 D , available in the online version). In user-centered design (UCD), we condensed the order set even further into fewer sections and made major changes to both the order set structure and the “prepare” and “transfuse” orders (see Supplementary Figs. S1 E and F and S2C and D , available in the online version). OR, operating room.

Fig. 2

Summative testing design. Participants (1) completed a quiz assessing their knowledge of appropriate special processing requests for blood products, (2) completed blood ordering tasks for 4 scenarios with either the original design or UCD orders and order set, (3) retook the same quiz with questions and answers shuffled, (4) completed one additional blood ordering scenario with the opposite design, and (5) answered survey questions on their preference between the designs and the perceived usability, ease of use, efficiency, and overall rating of the UCD. UCD, user-centered design.