Impact of Clinicians' Use of Electronic Knowledge Resources on Clinical and Learning Outcomes: Systematic Review and Meta-Analysis

Abstract

Background: Clinicians use electronic knowledge resources, such as Micromedex, UpToDate, and Wikipedia, to deliver evidence-based care and engage in point-of-care learning. Despite this use in clinical practice, their impact on patient care and learning outcomes is incompletely understood. A comprehensive synthesis of available evidence regarding the effectiveness of electronic knowledge resources would guide clinicians, health care system administrators, medical educators, and informaticians in making evidence-based decisions about their purchase, implementation, and use.

Objective: The aim of this review is to quantify the impact of electronic knowledge resources on clinical and learning outcomes.

Methods: We searched MEDLINE, Embase, PsycINFO, and the Cochrane Library for articles published from 1991 to 2017. Two authors independently screened studies for inclusion and extracted outcomes related to knowledge, skills, attitudes, behaviors, patient effects, and cost. We used random-effects meta-analysis to pool standardized mean differences (SMDs) across studies.

Results: Of 10,811 studies screened, we identified 25 eligible studies published between 2003 and 2016. A total of 5 studies were randomized trials, 22 involved physicians in practice or training, and 10 reported potential conflicts of interest. A total of 15 studies compared electronic knowledge resources with no intervention. Of these, 7 reported clinician behaviors, with a pooled SMD of 0.47 (95% CI 0.27 to 0.67; P<.001), and 8 reported objective patient effects with a pooled SMD of 0.19 (95% CI 0.07 to 0.32; P=.003). Heterogeneity was large (I²>50%) across studies. When compared with other resources-7 studies, not amenable to meta-analytic pooling-the use of electronic knowledge resources was associated with increased frequency of answering questions and perceived benefits on patient care, with variable impact on time to find an answer. A total of 2 studies compared different implementations of the same electronic knowledge resource.

Conclusions: Use of electronic knowledge resources is associated with a positive impact on clinician behaviors and patient effects. We found statistically significant associations between the use of electronic knowledge resources and improved clinician behaviors and patient effects. When compared with other resources, the use of electronic knowledge resources was associated with increased success in answering clinical questions, with variable impact on speed. Comparisons of different implementation strategies of the same electronic knowledge resource suggest that there are benefits from allowing clinicians to choose to access the resource, versus automated display of resource information, and from integrating patient-specific information. A total of 4 studies compared different commercial electronic knowledge resources, with variable results. Resource implementation strategies can significantly influence outcomes but few studies have examined such factors.

Keywords: clinical decision support; educational technology; health information technology; information systems; medical education.

Figure 1

Trial flowchart.

Figure 2

Comparative usage of electronic knowledge resources versus no intervention. Knowledge outcome analyses are weighted by user, while behavior and patient effects analyses are weighted by patients or hospitals. “a” denotes a locally developed resource; “b” is the number of hospitals, not patients; “c” indicates no comparison group (ie, one-group, pre-/postintervention study). Abx Guide: Johns Hopkins Antibiotic Guide; Ang Soft: angina software; CEM: clinical evidence module; eAAP: Emergency Asthma Action Plan; Epoc: Epocrates; GRAIDS: Genetic Risk Assessment on the Internet with Decision Support; InfoRet: InfoRetriever; MD: practicing physicians; MOC: Maintenance of Certification; MS: medical students; NP: nurse practitioners; ns: not specified; PG: residents; PIER: Physicians’ Information and Education Resource; Rep Sup: Report Support; SCAMP: Standardized Clinical Assessment and Management Plans; UTD: UpToDate.

Figure 3

Impact of electronic knowledge resources in comparison with other resources (Panel A) and alternate electronic knowledge resources (Panel B). All analyses are weighted by patients except as noted. “a” refers to analysis weighted by users; “b” means the comparison group (ie, study data) is the same for these contrasts; “c” means the comparison type “Mixed” indicates a comparison with both electronic and nonelectronic knowledge resources; “d” means the comparison type “Any other” indicates users could select any resource, except the ones it was being compared against; “e” denotes a locally developed resource. 5-min: 5-Minute Clinical Consult; AccessMed: AccessMedicine; ARUSC: Antibiotic Utilization and Surveillance-Control; Clin Evid: clinical evidence; Epoc: Epocrates; InfoRet: InfoRetriever; K: Knowledge; MD: practicing physicians; MMX: Micromedex; MS: medical students; NOS: not otherwise specified; NP: nurse practitioners; ns: not specified; PG: residents; Q: question; rec: recommendation; spec: specific; Taras: Tarascon Pharmacopeia; Trip: Turning Research Into Practice; UTD: UpToDate; Wiki: Wikipedia.