LLMonFHIR: A Physician-Validated, Large Language Model-Based Mobile Application for Querying Patient Electronic Health Data

Abstract

Background: To improve healthcare quality and empower patients, federal legislation requires nationwide interoperability of electronic health records (EHRs) through Fast Healthcare Interoperability Resources (FHIR) application programming interfaces. Nevertheless, key barriers to patient EHR access-limited functionality, English, and health literacy-persist, impeding equitable access to these benefits.

Objectives: This study aimed to develop and evaluate a digital health solution to address barriers preventing patient engagement with personal health information, focusing on individuals managing chronic cardiovascular conditions.

Methods: We present LLMonFHIR, an open-source mobile application that uses large language models (LLMs) to allow users to "interact" with their health records at any degree of complexity, in various languages, and with bidirectional text-to-speech functionality. In a pilot evaluation, physicians assessed LLMonFHIR responses to queries on 6 SyntheticMass FHIR patient datasets, rating accuracy, understandability, and relevance on a 5-point Likert scale.

Results: A total of 210 LLMonFHIR responses were evaluated by physicians, receiving high median scores for accuracy (5/5), understandability (5/5), and relevance (5/5). Challenges summarizing health conditions and retrieving lab results were noted, with variability in responses and occasional omissions underscoring the need for precise preprocessing of data.

Conclusions: LLMonFHIR's ability to generate responses in multiple languages and at varying levels of complexity, along with its bidirectional text-to-speech functionality, give it the potential to empower individuals with limited functionality, English, and health literacy to access the benefits of patient-accessible EHRs.

Keywords: artificial intelligence; digital health; large language model; literacy; mobile application.

Graphical abstract

Central Illustration

The LLMonFHIR iOS Application: LLM-Enabled FHIR Data Access This figure illustrates the workflow of the LLMonFHIR application, which enables patient interaction with their electronic health records (EHRs) using large language models (LLMs). The process begins with user questions (1), which are processed through a targeted function call for relevant Fast Healthcare Interoperability Resources (FHIR) (2). These resources, are retrieved from the health system's certified EHR system via FHIR endpoints using Apple HealthKit, include structured EHR data such as demographics, medications, and laboratory results. The retrieved FHIR resources (3) are then analyzed by a cloud-hosted LLM, such as OpenAI's GPT-4, to generate a tailored response. The final output is delivered as an AI agent response (4), providing users with comprehensible and actionable health insights. Arrows in the diagram represent the flow of information between the LLMonFHIR app, the EHR system, and the LLM.

Figure 1

The LLMonFHIR User Interface (UI) The LLMonFHIR mobile application provides an intuitive interface for patients to interact with their electronic health records (EHRs) using large language models (LLMs). (A) An overview of all available FHIR (Fast Healthcare Interoperability Resources)-standardized health records, enabling users to browse their data comprehensively. (B) The app's ability to localize, translate, and summarize a selected resource into patient-friendly language, shown here in German. (C) The interactive chat feature, which allows users to ask questions about their health data and receive tailored responses in natural language.

Figure 2

Physician Evaluation of LLMonFHIR Responses The figure illustrates the scoring of LLMonFHIR-generated responses by expert physician reviewers across 6 synthetic FHIR (Fast Healthcare Interoperability Resources) patient datasets. A total of 210 responses were evaluated for accuracy, understandability, and relevance, with median scores of 5, 5, and 5 out of 5, respectively. Score ranges are provided to indicate variability in scoring across questions and categories.

Figure 3

Example of Reviewed LLMonFHIR Outputs This figure illustrates examples of LLMonFHIR-generated responses to physician queries during the pilot evaluation, highlighting the model's strengths and limitations. (A) A response to Question 3 accurately identifies no allergies, demonstrating high accuracy without hallucinations. (B) A response to Question 2 provides a complete and relevant list of medications, showcasing clarity and precision. (C) A response to Question 5 includes verbose, generic suggestions, reflecting limited relevance and specificity. (D) A response to Question 7 fails to retrieve relevant laboratory values, underscoring challenges in resource identification and data preprocessing.