A Large Language Model-Based Approach for Coding Information from Free-Text Reported in Fall Risk Surveillance Systems: New Opportunities for In-Hospital Risk Management

Abstract

Background/Objectives: Falls are the most common adverse in-hospital event, resulting in a considerable social and economic burden on individuals, their families, and the healthcare system. This study aims to develop and implement an automatic coding system using large language models (LLMs) to extract and categorize free-text information (including the location of the fall and any resulting injury) from in-hospital fall records. Methods: The study used the narrative description of the falls reported through the Incident Reporting system to the Risk Management Service of an Italian Local Health Authority in Italy (name not disclosed as per research agreement). The OpenAI application programming interface (API) was used to access the generative pre-trained transformers (GPT) models, extract data from the narrative description of the falls, and perform the classification task. The GPT-4-turbo models were used for the classification task. Two independent reviewers manually coded the information, representing the gold standard for the classification task. Sensitivity, specificity, and accuracy were calculated to evaluate the performance of the task. Results: The analysis included 187 fall records with free-text event descriptions detailing the location of the fall and 93 records providing information about the presence or absence of an injury. GPT-4-turbo showed excellent performance, with specificity, sensitivity, and accuracy values of at least 0.913 for detecting the location and 0.953 for detecting the injury. Conclusions: The GPT models effectively extracted and categorized the information, even though the text was not optimized for GPT-based analysis. This shows their potential for the use of LLMs in clinical risk management research.

Keywords: free text; in-hospital falls; large language models; risk management.