Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Sep 15;8(3):e70011.
doi: 10.1002/oto2.70011. eCollection 2024 Jul-Sep.

Evaluating ChatGPT as a Patient Education Tool for COVID-19-Induced Olfactory Dysfunction

Elliott M Sina  1 , Daniel J Campbell  2 , Alexander Duffy  2 , Shreya Mandloi  2 , Peter Benedict  2 , Douglas Farquhar  2 , Aykut Unsal  2 , Gurston Nyquist  2
Affiliations

Evaluating ChatGPT as a Patient Education Tool for COVID-19-Induced Olfactory Dysfunction

Elliott M Sina et al. OTO Open. .

Abstract

Objective: While most patients with COVID-19-induced olfactory dysfunction (OD) recover spontaneously, those with persistent OD face significant physical and psychological sequelae. ChatGPT, an artificial intelligence chatbot, has grown as a tool for patient education. This study seeks to evaluate the quality of ChatGPT-generated responses for COVID-19 OD.

Study design: Quantitative observational study.

Setting: Publicly available online website.

Methods: ChatGPT (GPT-4) was queried 4 times with 30 identical questions. Prior to questioning, Chat-GPT was "prompted" to respond (1) to a patient, (2) to an eighth grader, (3) with references, and (4) no prompt. Answer accuracy was independently scored by 4 rhinologists using the Global Quality Score (GCS, range: 1-5). Proportions of responses at incremental score thresholds were compared using χ 2 analysis. Flesch-Kincaid grade level was calculated for each answer. Relationship between prompt type and grade level was assessed via analysis of variance.

Results: Across all graded responses (n = 480), 364 responses (75.8%) were "at least good" (GCS ≥ 4). Proportions of responses that were "at least good" (P < .0001) or "excellent" (GCS = 5) (P < .0001) differed by prompt; "at least moderate" (GCS ≥ 3) responses did not (P = .687). Eighth-grade level (14.06 ± 2.3) and patient-friendly (14.33 ± 2.0) responses were significantly lower mean grade level than no prompting (P < .0001).

Conclusion: ChatGPT provides appropriate answers to most questions on COVID-19 OD regardless of prompting. However, prompting influences response quality and grade level. ChatGPT responds at grade levels above accepted recommendations for presenting medical information to patients. Currently, ChatGPT offers significant potential for patient education as an adjunct to the conventional patient-physician relationship.

Keywords: AI hallucination; COVID‐19; ChatGPT; Flesch‐Kincaid grade level; anosmia; artificial intelligence; chatbot; olfactory dysfunction; patient education; prompting.

PubMed Disclaimer

Conflict of interest statement

None.

Figures

Figure 1
Figure 1
Colormap representation of average graded ChatGPT responses by prompt type. Average Global Quality Score for each question posed to ChatGPT.
Figure 2
Figure 2
Analysis of variance assessing prompt type by grade level and word count. *, **, ***, and ****P ≤ .05, .01, .001, and .0001, respectively.
See this image and copyright information in PMC

References

    1. Mitchell MB, Workman AD, Rathi VK, Bhattacharyya N. Smell and taste loss associated with COVID‐19 infection. Laryngoscope. 2023;133(9):2357‐2361. 10.1002/lary.30802 - DOI - PubMed
    1. Mehraeen E, Behnezhad F, Salehi MA, Noori T, Harandi H, SeyedAlinaghi S. Olfactory and gustatory dysfunctions due to the coronavirus disease (COVID‐19): a review of current evidence. Eur Arch Otrhinolaryngol. 2021;278(2):307‐312. 10.1007/s00405-020-06120-6 - DOI - PMC - PubMed
    1. Tan BKJ, Han R, Zhao JJ, et al. Prognosis and persistence of smell and taste dysfunction in patients with covid‐19: meta‐analysis with parametric cure modelling of recovery curves. BMJ. 2022;378:e069503. 10.1136/bmj-2021-069503 - DOI - PMC - PubMed
    1. Peterson AM, Miller BJ, Kallogjeri D, et al. Stellate ganglion block for the treatment of COVID‐19‐induced olfactory dysfunction: a prospective pilot study. Otolaryngol Head Neck Surg. 2024;170(1):272‐276. 10.1002/ohn.530 - DOI - PMC - PubMed
    1. Aaraj MA, Boorinie M, Salfity L, Eweiss A. The use of Platelet rich plasma in COVID‐19 induced olfactory dysfunction: systematic review. Indian J Otolaryngol Head Neck Surg. 2023;75:3093‐3097. 10.1007/s12070-023-03938-4 - DOI - PMC - PubMed

LinkOut - more resources