Readability of Orthopaedic Patient Educational Material: An artificial intelligence application

Abstract

Background: This study aims to determine the efficacy of the use of artificial intelligence (AI) in rewriting orthopaedic trauma hospital patient educational materials to a patient-appropriate reading level.

Materials and methods: 35 orthopaedic patient educational articles were identified from three hospital networks with Level 1 Trauma Centers, categorized based on average reading level. They were run through a formatting Python code, and then a secondary code to determine readability metrics outlined in Table 1. The articles were then rewritten via four iterations of Generative Pre-Trained Transformer (GPT) AI language models. Each model was given the same prompt, outlined in Fig. 1, to rewrite the articles to a 6th grade reading level per AMA recommendations. The rewritten articles were checked for accuracy and formatted and scored to determine mean reading level. Additional analysis was run comparing 9 different AI models from 3 different companies, using the same prompt, comparing cost and percent token reduction.

Results: All GPT AI models lowered the mean combined grade level (Table 2). Fig. 2 compares each GPT model's output to the original articles reading grade level. The oldest model (GPT-3.5-Turbo) was the least consistent and least effective. GPT-4o-Mini and GPT-4o were the most effective and consistent regardless of original article difficulty. Table 3 outlines the cost of running all 35 articles through each GPT model. The most accurate model (GPT-4o) was only $0.61; however, there was only a 0.421 % increase in effectiveness comparing GPT-4o vs. GPT-4o-Mini, at a 175.38 % increase in cost. All GPT rewritten articles were screened for accuracy and determined to have no falsified information or medical inaccuracies. Expanded analysis across 9 AI models is demonstrated in Fig. 4. Fig. 5 compares cost and percent token reduction.

Conclusion: AI is a viable option for reducing the reading difficulty of patient educational materials while maintaining accuracy. Of the models included for analysis, GPT-4o-Mini appears to be the most efficient language model when considering effectiveness, cost, and maintenance of the information included in the original articles.

Keywords: Artificial intelligence; Artificial intelligence software; Orthopedic surgery; Patient education; Patient education material; Readability; Reading.