Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Oct 26;10(10):e39616.
doi: 10.2196/39616.

Evaluating the Impact on Clinical Task Efficiency of a Natural Language Processing Algorithm for Searching Medical Documents: Prospective Crossover Study

Eunsoo H Park #  1   2 , Hannah I Watson #  2 , Felicity V Mehendale  3 , Alison Q O'Neil  2   4 ; Clinical Evaluators  5
Collaborators, Affiliations

Evaluating the Impact on Clinical Task Efficiency of a Natural Language Processing Algorithm for Searching Medical Documents: Prospective Crossover Study

Eunsoo H Park et al. JMIR Med Inform. .

Abstract

Background: Information retrieval (IR) from the free text within electronic health records (EHRs) is time consuming and complex. We hypothesize that natural language processing (NLP)-enhanced search functionality for EHRs can make clinical workflows more efficient and reduce cognitive load for clinicians.

Objective: This study aimed to evaluate the efficacy of 3 levels of search functionality (no search, string search, and NLP-enhanced search) in supporting IR for clinical users from the free text of EHR documents in a simulated clinical environment.

Methods: A clinical environment was simulated by uploading 3 sets of patient notes into an EHR research software application and presenting these alongside 3 corresponding IR tasks. Tasks contained a mixture of multiple-choice and free-text questions. A prospective crossover study design was used, for which 3 groups of evaluators were recruited, which comprised doctors (n=19) and medical students (n=16). Evaluators performed the 3 tasks using each of the search functionalities in an order in accordance with their randomly assigned group. The speed and accuracy of task completion were measured and analyzed, and user perceptions of NLP-enhanced search were reviewed in a feedback survey.

Results: NLP-enhanced search facilitated more accurate task completion than both string search (5.14%; P=.02) and no search (5.13%; P=.08). NLP-enhanced search and string search facilitated similar task speeds, both showing an increase in speed compared to the no search function, by 11.5% (P=.008) and 16.0% (P=.007) respectively. Overall, 93% of evaluators agreed that NLP-enhanced search would make clinical workflows more efficient than string search, with qualitative feedback reporting that NLP-enhanced search reduced cognitive load.

Conclusions: To the best of our knowledge, this study is the largest evaluation to date of different search functionalities for supporting target clinical users in realistic clinical workflows, with a 3-way prospective crossover study design. NLP-enhanced search improved both accuracy and speed of clinical EHR IR tasks compared to browsing clinical notes without search. NLP-enhanced search improved accuracy and reduced the number of searches required for clinical EHR IR tasks compared to direct search term matching.

Keywords: clinical decision support; clinical informatics; electronic health records; natural language processing; semantic search.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest: HIW and AQO are employees of Canon Medical Research Europe, who provided the software and algorithms for this evaluation. EHP was sponsored by Canon Medical Research Europe during her Spring 2021 BSc research project at the University of Edinburgh (“Evaluation of a natural language processing algorithm for searching medical documents”) which was the basis for this evaluation. EHP had previously performed paid annotation work for the development of the NLP-enhanced search tool.

Figures

Figure 1
Figure 1
Example results for (A) string search and (B) NLP-enhanced search for the search term “heart.” String search returned only direct matches to “heart” (green highlights) whereas NLP-enhanced search also returns semantically related terms (yellow highlights) such as the following: “coronary,” the misspelling of atrial (fibrillation) as “atriall,” and the appearance of “heart” within the abbreviation of heart failure, “HF.” NLP: natural language processing.
Figure 2
Figure 2
Study design. The 3 tasks were performed using a prospective crossover design, in which each group undertook the tasks in the same order with a predetermined order of the search intervention; the order was different for different groups. Finally, all evaluators were asked to fill in a review questionnaire. NLP: natural language processing.
Figure 3
Figure 3
Screenshot of the evaluation environment during a task. Evaluators only had permission to view the two relevant sites: the patient-centric viewer (left) and the evaluation platform (right). The patient-centric viewer contains the synthetic patient documents for a given patient (in this case “Joseph Williams”) with “hba1c” as the search term. The evaluation platform detailed the clinical scenarios, task-specific instructions, and question-and-answer sections.
See this image and copyright information in PMC

References

    1. Holanda A, do Carmo E Sá HL, Vieira A, Catrib AMF. Use and satisfaction with electronic health record by primary care physicians in a health district in Brazil. J Med Syst. 2012 Oct;36(5):3141–3149. doi: 10.1007/s10916-011-9801-3. doi: 10.1007/s10916-011-9801-3. - DOI - DOI - PubMed
    1. King J, Patel V, Jamoom E, Furukawa MF. Clinical benefits of electronic health record use: national findings. Health Serv Res. 2014 Feb;49(1 Pt 2):392–404. doi: 10.1111/1475-6773.12135. https://europepmc.org/abstract/MED/24359580 - DOI - PMC - PubMed
    1. Burke H, Sessums L, Hoang A, Becher DA, Fontelo P, Liu F, Stephens M, Pangaro LN, O'Malley PG, Baxi NS, Bunt CW, Capaldi VF, Chen JM, Cooper BA, Djuric DA, Hodge JA, Kane S, Magee C, Makary ZR, Mallory RM, Miller T, Saperstein A, Servey J, Gimbel RW. Electronic health records improve clinical note quality. J Am Med Inform Assoc. 2015 Jan 1;22(1):199–205. doi: 10.1136/amiajnl-2014-002726. https://europepmc.org/abstract/MED/25342178 amiajnl-2014-002726 - DOI - PMC - PubMed
    1. Entzeridou E, Markopoulou E, Mollaki V. Public and physician's expectations and ethical concerns about electronic health record: benefits outweigh risks except for information security. Int J Med Inform. 2018 Feb;110:98–107. doi: 10.1016/j.ijmedinf.2017.12.004. doi: 10.1016/j.ijmedinf.2017.12.004.S1386-5056(17)30447-1 - DOI - DOI - PubMed
    1. Kroth P, Morioka-Douglas N, Veres S, Babbott S, Poplau S, Qeadan F, Parshall C, Corrigan K, Linzer M. Association of electronic health record design and use factors with clinician stress and burnout. JAMA Netw Open. 2019 Aug 02;2(8):e199609. doi: 10.1001/jamanetworkopen.2019.9609. https://jamanetwork.com/journals/jamanetworkopen/fullarticle/10.1001/jam... 2748054 - DOI - DOI - PMC - PubMed

LinkOut - more resources