The potential of Generative Pre-trained Transformer 4 (GPT-4) to analyse medical notes in three different languages: a retrospective model-evaluation study

Abstract

Background: Patient notes contain substantial information but are difficult for computers to analyse due to their unstructured format. Large-language models (LLMs), such as Generative Pre-trained Transformer 4 (GPT-4), have changed our ability to process text, but we do not know how effectively they handle medical notes. We aimed to assess the ability of GPT-4 to answer predefined questions after reading medical notes in three different languages.

Methods: For this retrospective model-evaluation study, we included eight university hospitals from four countries (ie, the USA, Colombia, Singapore, and Italy). Each site submitted seven de-identified medical notes related to seven separate patients to the coordinating centre between June 1, 2023, and Feb 28, 2024. Medical notes were written between Feb 1, 2020, and June 1, 2023. One site provided medical notes in Spanish, one site provided notes in Italian, and the remaining six sites provided notes in English. We included admission notes, progress notes, and consultation notes. No discharge summaries were included in this study. We advised participating sites to choose medical notes that, at time of hospital admission, were for patients who were male or female, aged 18-65 years, had a diagnosis of obesity, had a diagnosis of COVID-19, and had submitted an admission note. Adherence to these criteria was optional and participating sites randomly chose which medical notes to submit. When entering information into GPT-4, we prepended each medical note with an instruction prompt and a list of 14 questions that had been chosen a priori. Each medical note was individually given to GPT-4 in its original language and in separate sessions; the questions were always given in English. At each site, two physicians independently validated responses by GPT-4 and responded to all 14 questions. Each pair of physicians evaluated responses from GPT-4 to the seven medical notes from their own site only. Physicians were not masked to responses from GPT-4 before providing their own answers, but were masked to responses from the other physician.

Findings: We collected 56 medical notes, of which 42 (75%) were in English, seven (13%) were in Italian, and seven (13%) were in Spanish. For each medical note, GPT-4 responded to 14 questions, resulting in 784 responses. In 622 (79%, 95% CI 76-82) of 784 responses, both physicians agreed with GPT-4. In 82 (11%, 8-13) responses, only one physician agreed with GPT-4. In the remaining 80 (10%, 8-13) responses, neither physician agreed with GPT-4. Both physicians agreed with GPT-4 more often for medical notes written in Spanish (86 [88%, 95% CI 79-93] of 98 responses) and Italian (82 [84%, 75-90] of 98 responses) than in English (454 [77%, 74-80] of 588 responses).

Interpretation: The results of our model-evaluation study suggest that GPT-4 is accurate when analysing medical notes in three different languages. In the future, research should explore how LLMs can be integrated into clinical workflows to maximise their use in health care.

Funding: None.

Figure 1:. Study design

(A) Study processes. (B) Instruction prompts given to GPT-4 with 14 questions chosen by our study group before submission of medical notes. In the prompt, the term medical_note was replaced by the actual medical note. Please note that questions and prompts are verbatim questions and prompts asked to GPT-4, and have not been edited. GPT-4=Generative Pre-trained Transformer 4. USPSTF=US Preventive Services Task Force.

Figure 2:. Clinical validation of responses from GPT-4

(A) Agreement of two separate physicians with GPT-4. (B) Agreement with responses by site. (C) Agreement with responses by language. (D) Agreement with results by question. The full list of questions is provided in figure 1. Both agreed refers to when both physicians answered Yes to the question “Do you agree with GPT-4’s answer?” after reading its response. One agreed refers to when one physician agreed with the response from GPT-4, but the other did not. Neither agreed refers to when both physicians did not agree with the response from GPT-4. GPT-4=Generative Pre-trained Transformer 4. *p<0·0001. †p=0·0073.

Figure 3:. Agreement between physicians and GPT-4

Nodes do not sum to 100% due to rounding. Both agreed refers to when both physicians answered Yes to the question “Do you agree with GPT-4’s answer?” after reading its response. One agreed refers to when one physician agreed with the response from GPT-4, but the other did not. Neither agreed refers to when both physicians did not agree with the response from GPT-4. GPT-4=Generative Pre-trained Transformer 4.

Figure 4:. Ability of GPT-4 to select patients for hypothetical study enrolment

(A) Inclusion criteria. We only included questions for which both physicians agreed or disagreed with GPT-4 in this analysis. Please note that questions are verbatim questions asked to GPT-4 and to physicians, and have not been edited. (B) Sensitivity and specificity, by inclusion criteria. Denominators for each criterion are numerators for that criterion in the column showing agreement between physicians in panel A. For example, we considered 51 notes in the analysis for age, 34 of which were within the age range of the study and were used to calculate sensitivity and 17 of which were not within the age range of the study and were used to calculate specificity. (C) Proportion of times GPT-4 was able to correctly provide all four inclusion criteria in a single patient. (D) Proportion of times GPT-4 was able to correctly provide all three inclusion criteria (excluding admission note) in a single patient. GPT-4=Generative Pre-trained Transformer 4. NA=not applicable.