Assessing the Efficacy of Large Language Models in Health Literacy: A Comprehensive Cross-Sectional Study

Abstract

Enhanced health literacy in children has been empirically linked to better health outcomes over the long term; however, few interventions have been shown to improve health literacy. In this context, we investigate whether large language models (LLMs) can serve as a medium to improve health literacy in children. We tested pediatric conditions using 26 different prompts in ChatGPT-3.5, ChatGPT-4, Microsoft Bing, and Google Bard (now known as Google Gemini). The primary outcome measurement was the reading grade level (RGL) of output as assessed by Gunning Fog, Flesch-Kincaid Grade Level, Automated Readability Index, and Coleman-Liau indices. Word counts were also assessed. Across all models, output for basic prompts such as "Explain" and "What is (are)," were at, or exceeded, the tenth-grade RGL. When prompts were specified to explain conditions from the first- to twelfth-grade level, we found that LLMs had varying abilities to tailor responses based on grade level. ChatGPT-3.5 provided responses that ranged from the seventh-grade to college freshmen RGL while ChatGPT-4 outputted responses from the tenth-grade to the college senior RGL. Microsoft Bing provided responses from the ninth- to eleventh-grade RGL while Google Bard provided responses from the seventh- to tenth-grade RGL. LLMs face challenges in crafting outputs below a sixth-grade RGL. However, their capability to modify outputs above this threshold, provides a potential mechanism for adolescents to explore, understand, and engage with information regarding their health conditions, spanning from simple to complex terms. Future studies are needed to verify the accuracy and efficacy of these tools.

Keywords: Artificial Intelligence; ChatGPT; Google Bard; Google Gemini; Health Literacy; Large Language Models; Microsoft Bing; Pediatrics; Reading Grade Level.

Figure 1

Reading grade levels for basic prompts. Legend: Basic Prompts P0 “What is (are) {medical condition}” and P1 “Explain {medical condition}” were tested through the LLMs. The aRGL of outputs are shown. *, **, ***, **** correspond to p<0.05, p<0.01, p<0.001, and p<0.0001, respectively. Comparisons between LLM for identical prompts are not shown, but all differences are statistically significant p<0.0001, except between ChatGPT-3.5 and ChatGPT-4 for “what is” / “what are.”

Figure 2

Reading grade level of output after running “Explain {} to a ____ grader” through each LLM. Legend: Each LLM was asked, “Explain {medical condition} to a __ grader.” First- through twelfth-grade were tested by filling in the blank. A. The aRGL of outputs is depicted for each LLM. From top to bottom, GPT-3.5, GPT-4, Bing, and Bard are depicted. B. Grade-level outputs for each LLM from panel A are set side to side for comparison between LLMs.

Figure 3

Reading grade level of output after running “Explain {} at a ____- grade reading level” through each LLM. Legend: Each LLM was asked “Explain {medical condition} at a __-grade reading level.” First- through twelfth-grade were tested by filling in the blank. A. The aRGL of outputs is depicted for each LLM. From top to bottom, GPT-3.5, GPT-4, Bing, and Bard are depicted. B. Grade-level outputs for each LLM from panel A are set side to side for comparison between LLMs.