Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Aug 16:10:e59213.
doi: 10.2196/59213.

A Language Model-Powered Simulated Patient With Automated Feedback for History Taking: Prospective Study

Friederike Holderried  1 , Christian Stegemann-Philipps  1 , Anne Herrmann-Werner  1 , Teresa Festl-Wietek  1 , Martin Holderried  2 , Carsten Eickhoff  3 , Moritz Mahling  1   2
Affiliations

A Language Model-Powered Simulated Patient With Automated Feedback for History Taking: Prospective Study

Friederike Holderried et al. JMIR Med Educ. .

Abstract

Background: Although history taking is fundamental for diagnosing medical conditions, teaching and providing feedback on the skill can be challenging due to resource constraints. Virtual simulated patients and web-based chatbots have thus emerged as educational tools, with recent advancements in artificial intelligence (AI) such as large language models (LLMs) enhancing their realism and potential to provide feedback.

Objective: In our study, we aimed to evaluate the effectiveness of a Generative Pretrained Transformer (GPT) 4 model to provide structured feedback on medical students' performance in history taking with a simulated patient.

Methods: We conducted a prospective study involving medical students performing history taking with a GPT-powered chatbot. To that end, we designed a chatbot to simulate patients' responses and provide immediate feedback on the comprehensiveness of the students' history taking. Students' interactions with the chatbot were analyzed, and feedback from the chatbot was compared with feedback from a human rater. We measured interrater reliability and performed a descriptive analysis to assess the quality of feedback.

Results: Most of the study's participants were in their third year of medical school. A total of 1894 question-answer pairs from 106 conversations were included in our analysis. GPT-4's role-play and responses were medically plausible in more than 99% of cases. Interrater reliability between GPT-4 and the human rater showed "almost perfect" agreement (Cohen κ=0.832). Less agreement (κ<0.6) detected for 8 out of 45 feedback categories highlighted topics about which the model's assessments were overly specific or diverged from human judgement.

Conclusions: The GPT model was effective in providing structured feedback on history-taking dialogs provided by medical students. Although we unraveled some limitations regarding the specificity of feedback for certain feedback categories, the overall high agreement with human raters suggests that LLMs can be a valuable tool for medical education. Our findings, thus, advocate the careful integration of AI-driven feedback mechanisms in medical training and highlight important aspects when LLMs are used in that context.

Keywords: ChatGPT; GPT: LLM; LLMs; NLP; TEL; artificial intelligence; chatbot; chatbots; communication; communication skills; conversational agent; conversational agents; histories; history; interaction; interactions; language model; language models; machine learning; medical education; natural language processing; relationship; relationships; simulated; student; students; technology enhanced education; virtual patients communication.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest: None declared.

Figures

Figure 1
Figure 1
Screenshot of the chatbot interface as presented to participants (translated from German): (A) the interface during the interactive dialog and (B) the interface presenting the feedback.
Figure 2
Figure 2
Heat map showing the percentage of conversations mentioning the feedback categories for both raters: Generative Pretrained Transformer (GPT) in the first column, and human rater in the second.
Figure 3
Figure 3
Cohen κ for every category of feedback for the human rater and Generative Pretrained Transformer (GPT) as a rater, with the different feedback topics displayed in different colors.
See this image and copyright information in PMC

References

    1. Hampton JR, Harrison MJ, Mitchell JR, Prichard JS, Seymour C. Relative contributions of history-taking, physical examination, and laboratory investigation to diagnosis and management of medical outpatients. Br Med J. 1975;2(5969):486–489. doi: 10.1136/bmj.2.5969.486. https://europepmc.org/abstract/MED/1148666 - DOI - PMC - PubMed
    1. Peterson MC, Holbrook JH, Von Hales D, Smith NL, Staker LV. Contributions of the history, physical examination, and laboratory investigation in making medical diagnoses. West J Med. 1992;156(2):163–165. https://europepmc.org/abstract/MED/1536065 - PMC - PubMed
    1. Dorr Goold S, Lipkin M. The doctor-patient relationship: challenges, opportunities, and strategies. J Gen Intern Med. 1999;14(Suppl 1):S26–S33. doi: 10.1046/j.1525-1497.1999.00267.x. https://europepmc.org/abstract/MED/9933492 - DOI - PMC - PubMed
    1. Hausberg MC, Hergert A, Kröger C, Bullinger M, Rose M, Andreas S. Enhancing medical students' communication skills: development and evaluation of an undergraduate training program. BMC Med Educ. 2012;12:16. doi: 10.1186/1472-6920-12-16. https://bmcmededuc.biomedcentral.com/articles/10.1186/1472-6920-12-16 1472-6920-12-16 - DOI - PMC - PubMed
    1. Deveugele M, Derese A, De Maesschalck S, Willems S, Van Driel M, De Maeseneer J. Teaching communication skills to medical students, a challenge in the curriculum? Patient Educ Couns. 2005;58(3):265–270. doi: 10.1016/j.pec.2005.06.004.S0738-3991(05)00171-0 - DOI - PubMed

LinkOut - more resources