Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
[Preprint]. 2023 Sep 4:2023.08.31.23294924.
doi: 10.1101/2023.08.31.23294924.

A comparison of large language model versus manual chart review for extraction of data elements from the electronic health record

Jin Ge  1 , Michael Li  1 , Molly B Delk  2 , Jennifer C Lai  1
Affiliations

A comparison of large language model versus manual chart review for extraction of data elements from the electronic health record

Jin Ge et al. medRxiv. .

Update in

Abstract

Importance: Large language models (LLMs) have proven useful for extracting data from publicly available sources, but their uses in clinical settings and with clinical data are unknown.

Objective: To determine the accuracy of data extraction using "Versa Chat," a chat implementation of the general-purpose OpenAI gpt-35-turbo LLM model, versus manual chart review for hepatocellular carcinoma (HCC) imaging reports.

Design: We engineered a prompt for the data extraction task of six distinct data elements and input 182 abdominal imaging reports that were also manually tagged. We evaluated performance by calculating accuracy, precision, recall, and F1 scores.

Setting/participants: Cross-sectional abdominal imaging reports of patients diagnosed with hepatocellular carcinoma enrolled in the Functional Assessment in Liver Transplantation (FrAILT) study.

PubMed Disclaimer

Conflict of interest statement

Disclosures: The authors of this manuscript have the following potential conflicts of interest to disclose: Dr. Jin Ge receives research support from Merck and Co; and consults for Astellas Pharmaceuticals/Iota Biosciences.Dr. Jennifer C. Lai receives research support from Lipocene and Vir Biotechnologies; receives an education grant from Nestle Nutrition Sciences; serves on an advisory board for Novo Nordisk; and consults for Genfit, Third Rock Ventures, and Boehringer Ingelheim.

Figures

Figure 1 –
Figure 1 –
Final prompt used for data extraction from “Versa Chat” (gpt-35-turbo)
Figure 2 –
Figure 2 –
Example of an exchange with “Versa Chat” (gpt-35-turbo) using mock data
See this image and copyright information in PMC

References

    1. Ge J, Lai JC. Artificial intelligence-based text generators in hepatology: ChatGPT is just the beginning. Hepatol Commun. 2023;7(4). doi:10.1097/HC9.0000000000000097 - DOI - PMC - PubMed
    1. Singhal K, Azizi S, Tu T, et al. Large language models encode clinical knowledge. Nature. 2023;620(7972):172–180. doi:10.1038/s41586-023-06291-2 - DOI - PMC - PubMed
    1. Chernyak V, Fowler KJ, Kamaya A, et al. Liver Imaging Reporting and Data System (LI-RADS) Version 2018: Imaging of Hepatocellular Carcinoma in At-Risk Patients. Radiology. 2018;289(3):816–830. doi:10.1148/radiol.2018181494 - DOI - PMC - PubMed
    1. Azure OpenAI Service – Large Language Models for Generative AI. https://azure.microsoft.com/en-us/products/ai-services/openai-service-b. Accessed August 25, 2023.
    1. Azure OpenAI Service models - Azure OpenAI | Microsoft Learn. https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models. Accessed August 26, 2023.

Publication types