Studying the human-computer-terminology interface

Abstract

Objective: To explore the use of an observational, cognitive-based approach for differentiating between successful, suboptimal, and failed entry of coded data by clinicians in actual practice, and to detect whether causes for unsuccessful attempts to capture true intended meaning were due to terminology content, terminology representation, or user interface problems.

Design: Observational study with videotaping and subsequent coding of data entry events in an outpatient clinic at New York Presbyterian Hospital.

Participants: Eight attending physicians, 18 resident physicians, and 1 nurse practitioner, using the Medical Entities Dictionary (MED) to record patient problems, medications, and adverse reactions in an outpatient medical record system.

Measurements: Classification of data entry events as successful, suboptimal, or failed, and estimation of cause; recording of system response time and total event time.

Results: Two hundred thirty-eight data entry events were analyzed; 71.0 percent were successful, 6.3 percent suboptimal, and 22.7 percent failed; unsuccessful entries were due to problems with content in 13.0 percent of events, representation problems in 10.1 percent of events, and usability problems in 5.9 percent of events. Response time averaged 0.74 sec, and total event time averaged 40.4 sec. Of an additional 209 tasks related to drug dose and frequency terms, 94 percent were successful, 0.5 percent were suboptimal, and 6 percent failed, for an overall success rate of 82 percent.

Conclusions: Data entry by clinicians using the outpatient system and the MED was generally successful and efficient. The cognitive-based observational approach permitted detection of false-positive (suboptimal) and false-negative (failed due to user interface) data entry.

Figure 1

Sample DOP screen. Items in the “Adverse Reactions,” “Current Medications,” and Active Problems” windows were entered by clinicians using the controlled terminology.

Figure 2

Data entry for a patient problem. The user has typed “chf,” and the system has returned 13 terms.

Figure 3

Data entry for a patient prescription. The user has typed “PO” in the dose route field, and the system has recognized it as “by mouth.” The user is now selecting the does frequency from the terminology subset using apull-down menu.

Figure 4

Portable usability laboratory components. The video converter is placed between the computer and the monitor at the user's workstation. The converter sends an NTSC video signal via an S-video cable to the VCR on the portable usability laboratory cart. A microphone is used to provide an audio recording of the user, captured on the VCR and a cassette recorder. A laptop computer on the cart communicates with the video converter via an RS-232 cable. These three wires are bundled together to extend a maximum of 75 feet. The researchser can remain with the cart in the next room, observing the recording with a television monitor and headphones and controlling the panning and zooming functions the video converter.

Figure 5

Examples of video transcription and coding. The left column shows transcription of audio portion of videotape, with timing marks and some statement about user interaction with the computer. The right column shows the coding of the interaction while reviewing the videotape.