. 2017 Mar:67:1-10.

doi: 10.1016/j.jbi.2017.01.013. Epub 2017 Jan 25.

Development and empirical user-centered evaluation of semantically-based query recommendation for an electronic health record search engine

David A Hanauer¹, Danny T Y Wu², Lei Yang³, Qiaozhu Mei⁴, Katherine B Murkowski-Steffy⁵, V G Vinod Vydiswaran⁶, Kai Zheng⁷

Affiliations

¹ Department of Pediatrics, University of Michigan Medical School, 5312 CC, SPC 5940, 1500 East Medical Center Drive, Ann Arbor, MI 48109, USA; School of Information, University of Michigan, 105 South State Street, Ann Arbor, MI 48109, USA. Electronic address: hanauer@med.umich.edu.
² School of Information, University of Michigan, 105 South State Street, Ann Arbor, MI 48109, USA; Department of Pediatrics, University of Michigan Medical School, 5312 CC, SPC 5940, 1500 East Medical Center Drive, Ann Arbor, MI 48109, USA. Electronic address: tzuyu@umich.edu.
³ School of Information, University of Michigan, 105 South State Street, Ann Arbor, MI 48109, USA. Electronic address: yangle@umich.edu.
⁴ School of Information, University of Michigan, 105 South State Street, Ann Arbor, MI 48109, USA; Department of Electrical Engineering and Computer Science, University of Michigan, 2260 Hayward Street, Ann Arbor, MI 48109, USA. Electronic address: qmei@umich.edu.
⁵ Department of Health Management and Policy, School of Public Health, 1415 Washington Heights, Ann Arbor, MI 48109, USA. Electronic address: kbmurko@umich.edu.
⁶ Department of Learning Health Sciences, University of Michigan Medical School, 1111 East Catherine Street, Ann Arbor, MI 48109, USA; School of Information, University of Michigan, 105 South State Street, Ann Arbor, MI 48109, USA. Electronic address: vgvinodv@med.umich.edu.
⁷ Department of Health Management and Policy, School of Public Health, 1415 Washington Heights, Ann Arbor, MI 48109, USA; School of Information, University of Michigan, 105 South State Street, Ann Arbor, MI 48109, USA. Electronic address: kzheng@umich.edu.

PMID: 28131722
PMCID: PMC5378386
DOI: 10.1016/j.jbi.2017.01.013

Development and empirical user-centered evaluation of semantically-based query recommendation for an electronic health record search engine

David A Hanauer et al. J Biomed Inform. 2017 Mar.

. 2017 Mar:67:1-10.

doi: 10.1016/j.jbi.2017.01.013. Epub 2017 Jan 25.

Authors

David A Hanauer¹, Danny T Y Wu², Lei Yang³, Qiaozhu Mei⁴, Katherine B Murkowski-Steffy⁵, V G Vinod Vydiswaran⁶, Kai Zheng⁷

Affiliations

¹ Department of Pediatrics, University of Michigan Medical School, 5312 CC, SPC 5940, 1500 East Medical Center Drive, Ann Arbor, MI 48109, USA; School of Information, University of Michigan, 105 South State Street, Ann Arbor, MI 48109, USA. Electronic address: hanauer@med.umich.edu.
² School of Information, University of Michigan, 105 South State Street, Ann Arbor, MI 48109, USA; Department of Pediatrics, University of Michigan Medical School, 5312 CC, SPC 5940, 1500 East Medical Center Drive, Ann Arbor, MI 48109, USA. Electronic address: tzuyu@umich.edu.
³ School of Information, University of Michigan, 105 South State Street, Ann Arbor, MI 48109, USA. Electronic address: yangle@umich.edu.
⁴ School of Information, University of Michigan, 105 South State Street, Ann Arbor, MI 48109, USA; Department of Electrical Engineering and Computer Science, University of Michigan, 2260 Hayward Street, Ann Arbor, MI 48109, USA. Electronic address: qmei@umich.edu.
⁵ Department of Health Management and Policy, School of Public Health, 1415 Washington Heights, Ann Arbor, MI 48109, USA. Electronic address: kbmurko@umich.edu.
⁶ Department of Learning Health Sciences, University of Michigan Medical School, 1111 East Catherine Street, Ann Arbor, MI 48109, USA; School of Information, University of Michigan, 105 South State Street, Ann Arbor, MI 48109, USA. Electronic address: vgvinodv@med.umich.edu.
⁷ Department of Health Management and Policy, School of Public Health, 1415 Washington Heights, Ann Arbor, MI 48109, USA; School of Information, University of Michigan, 105 South State Street, Ann Arbor, MI 48109, USA. Electronic address: kzheng@umich.edu.

PMID: 28131722
PMCID: PMC5378386
DOI: 10.1016/j.jbi.2017.01.013

Abstract

Objective: The utility of biomedical information retrieval environments can be severely limited when users lack expertise in constructing effective search queries. To address this issue, we developed a computer-based query recommendation algorithm that suggests semantically interchangeable terms based on an initial user-entered query. In this study, we assessed the value of this approach, which has broad applicability in biomedical information retrieval, by demonstrating its application as part of a search engine that facilitates retrieval of information from electronic health records (EHRs).

Materials and methods: The query recommendation algorithm utilizes MetaMap to identify medical concepts from search queries and indexed EHR documents. Synonym variants from UMLS are used to expand the concepts along with a synonym set curated from historical EHR search logs. The empirical study involved 33 clinicians and staff who evaluated the system through a set of simulated EHR search tasks. User acceptance was assessed using the widely used technology acceptance model.

Results: The search engine's performance was rated consistently higher with the query recommendation feature turned on vs. off. The relevance of computer-recommended search terms was also rated high, and in most cases the participants had not thought of these terms on their own. The questions on perceived usefulness and perceived ease of use received overwhelmingly positive responses. A vast majority of the participants wanted the query recommendation feature to be available to assist in their day-to-day EHR search tasks.

Discussion and conclusion: Challenges persist for users to construct effective search queries when retrieving information from biomedical documents including those from EHRs. This study demonstrates that semantically-based query recommendation is a viable solution to addressing this challenge.

Keywords: Electronic health records (E05.318.308.940.968.625.500); Information retrieval systems (L01.700.508.300); Query expansion; Query recommendation; Search engine (L01.470.875); Unified Medical Language System (L01.453.245.945.800).

Published by Elsevier Inc.

PubMed Disclaimer

Figures

**Figure 1. Components and Information Flow of the Query Recommendation Algorithm**
EHR documents (a) are matched to UMLS terms using the output of the MetaMap natural language processing software as well as to terms in a locally developed empiric synonym set (b, c). These document terms are then indexed using Lemur (d, e). In a similar manner, with the query suggestion feature turned on, user-supplied queries (j) are also matched to UMLS and ESS terms (k, l), but generic UMLS semantic types are removed (m) to provide matching on the more relevant clinical concepts. This results in an expanded query (n) that is compared to the index (e) for subsequent document ranking (f) and presentation to the user (g). When the query suggestion feature was turned off (h), parsed queries (l) are compared directly with the parsed terms in the index (e) without any synonym expansion. When the query suggestion was turned on (i), the parsed terms were expanded using the concepts to which they mapped. In either mode (on or off) the user could revise their queries (o) and repeat their search as many times as desired.

Figure 2. The Main User Interface of EHR-SE, with the Query Recommendation Feature Turned *Off*

**Figure 3. The Main User Interface of EHR-SE, with the Query Recommendation Feature Turned On**

See this image and copyright information in PMC

Cited by

Personal Health Information Recommender: implementing a tool for the empowerment of cancer patients.
Iatraki G, Kondylakis H, Koumakis L, Chatzimina M, Kazantzaki E, Marias K, Tsiknakis M. Iatraki G, et al. Ecancermedicalscience. 2018 Jul 11;12:851. doi: 10.3332/ecancer.2018.851. eCollection 2018. Ecancermedicalscience. 2018. PMID: 30079113 Free PMC article.
Extracting similar terms from multiple EMR-based semantic embeddings to support chart reviews.
Ye C, Fabbri D. Ye C, et al. J Biomed Inform. 2018 Jul;83:63-72. doi: 10.1016/j.jbi.2018.05.014. Epub 2018 May 22. J Biomed Inform. 2018. PMID: 29793071 Free PMC article.
A relevance and quality-based ranking algorithm applied to evidence-based medicine.
Serrano-Guerrero J, Romero FP, Olivas JA. Serrano-Guerrero J, et al. Comput Methods Programs Biomed. 2020 Jul;191:105415. doi: 10.1016/j.cmpb.2020.105415. Epub 2020 Feb 24. Comput Methods Programs Biomed. 2020. PMID: 32114416 Free PMC article.
Design, Implementation, and Usability of the Electronic Medical Record Search Engine (EMERSE) Tool.
Reyes CN, Zheng K, Hanauer DA. Reyes CN, et al. AMIA Annu Symp Proc. 2023 Apr 29;2022:932-941. eCollection 2022. AMIA Annu Symp Proc. 2023. PMID: 37128440 Free PMC article.
Google Medical Update: Why Is the Search Engine Decreasing Visibility of Health and Medical Information Websites?
Strzelecki A. Strzelecki A. Int J Environ Res Public Health. 2020 Feb 12;17(4):1160. doi: 10.3390/ijerph17041160. Int J Environ Res Public Health. 2020. PMID: 32059576 Free PMC article.

See all "Cited by" articles

References

1. Adler-Milstein J, DesRoches CM, Furukawa MF, et al. More Than Half of US Hospitals Have At Least A Basic EHR, But Stage 2 Criteria Remain Challenging For Most. Health Aff (Millwood) 2014;33(9):1664–1671. - PubMed
1. Berner ES, Moss J. Informatics challenges for the impending patient information explosion. J Am Med Inform Assoc. 2005;12(6):614–617. - PMC - PubMed
1. Biron P, Metzger MH, Pezet C, Sebban C, Barthuet E, Durand T. An information retrieval system for computerized patient records in the context of a daily hospital practice: the example of the Leon Berard Cancer Center (France) Appl Clin Inform. 2014;5(1):191–205. - PMC - PubMed
1. Gregg W, Jirjis J, Lorenzi NM, Giuse D. StarTracker: an integrated, web-based clinical search engine. AMIA Annu Symp Proc. 2003;855 - PMC - PubMed
1. Hanauer DA, Mei Q, Law J, Khanna R, Zheng K. Supporting information retrieval from electronic health records: A report of University of Michigan's nine-year experience in developing and using the Electronic Medical Record Search Engine (EMERSE) J Biomed Inform. 2015;55:290–300. - PMC - PubMed

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions

Grants and funding

LinkOut - more resources

Full Text Sources
Other Literature Sources
- scite Smart Citations

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Development and empirical user-centered evaluation of semantically-based query recommendation for an electronic health record search engine

Affiliations

Development and empirical user-centered evaluation of semantically-based query recommendation for an electronic health record search engine

Authors

Affiliations

Abstract

Figures

Similar articles

Cited by

References

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources

Other Literature Sources