Review of health information technology usability study methodologies

Abstract

Usability factors are a major obstacle to health information technology (IT) adoption. The purpose of this paper is to review and categorize health IT usability study methods and to provide practical guidance on health IT usability evaluation. 2025 references were initially retrieved from the Medline database from 2003 to 2009 that evaluated health IT used by clinicians. Titles and abstracts were first reviewed for inclusion. Full-text articles were then examined to identify final eligibility studies. 629 studies were categorized into the five stages of an integrated usability specification and evaluation framework that was based on a usability model and the system development life cycle (SDLC)-associated stages of evaluation. Theoretical and methodological aspects of 319 studies were extracted in greater detail and studies that focused on system validation (SDLC stage 2) were not assessed further. The number of studies by stage was: stage 1, task-based or user-task interaction, n=42; stage 2, system-task interaction, n=310; stage 3, user-task-system interaction, n=69; stage 4, user-task-system-environment interaction, n=54; and stage 5, user-task-system-environment interaction in routine use, n=199. The studies applied a variety of quantitative and qualitative approaches. Methodological issues included lack of theoretical framework/model, lack of details regarding qualitative study approaches, single evaluation focus, environmental factors not evaluated in the early stages, and guideline adherence as the primary outcome for decision support system evaluations. Based on the findings, a three-level stratified view of health IT usability evaluation is proposed and methodological guidance is offered based upon the type of interaction that is of primary interest in the evaluation.

Figure 1

Data management flowchart. In stage 2, system validation was performed by sensitivity and specificity testing, receiver operating characteristic curve, or observer variation and can be identified at title and abstract level with MeSH search. We identified 27 articles from 346 full-text articles and 283 articles at the title and abstract level.

Figure 2

A stratified view of health information technology evaluation. Level 1 targets system specification to understand user-task interaction for system development. Level 2 examines task performance to assess system validation and human–computer interaction. Level 3 aims to incorporate environmental factors to identify work processes and system impacts in real settings. Task/expectation complexity, user variances, and organizational support are factors that influence the use of the system, but are not problems of the system itself, and need to be differentiated from system-related issues.