Performance Assessment of ChatGPT versus Bard in Detecting Alzheimer's Dementia

Abstract

Large language models (LLMs) find increasing applications in many fields. Here, three LLM chatbots (ChatGPT-3.5, ChatGPT-4, and Bard) are assessed in their current form, as publicly available, for their ability to recognize Alzheimer's dementia (AD) and Cognitively Normal (CN) individuals using textual input derived from spontaneous speech recordings. A zero-shot learning approach is used at two levels of independent queries, with the second query (chain-of-thought prompting) eliciting more detailed information than the first. Each LLM chatbot's performance is evaluated on the prediction generated in terms of accuracy, sensitivity, specificity, precision, and F1 score. LLM chatbots generated a three-class outcome ("AD", "CN", or "Unsure"). When positively identifying AD, Bard produced the highest true-positives (89% recall) and highest F1 score (71%), but tended to misidentify CN as AD, with high confidence (low "Unsure" rates); for positively identifying CN, GPT-4 resulted in the highest true-negatives at 56% and highest F1 score (62%), adopting a diplomatic stance (moderate "Unsure" rates). Overall, the three LLM chatbots can identify AD vs. CN, surpassing chance-levels, but do not currently satisfy the requirements for clinical application.

Keywords: Alzheimer’s dementia; Bard; ChatGPT; GPT-3.5; GPT-4; Large Language Models; chain-of-thought; chatbots; ecological diagnostic screening; spontaneous speech; zero-shot learning.

Figure 1

Experimental methodology illustrating LLM chatbot zero-shot learning for predicting the ADReSSo speech recordings (71 testing set recordings 36 CN, 35 AD) as “AD”, “CN”, and “Unsure”.

Figure 2

Summary matrices (True vs. Predicted) of three LLM chatbots studied for Query 1 (Q1) and Query 2 (Q2), showing cognitive classification outcomes for three class prediction (AD vs. CN vs. Unsure) and their occurrence rate (%) when presented with the same text dataset from AD and CN subjects (True Class, 35 and 36 subjects, respectively).

Figure 3

Word cloud summaries for correct AD prediction (left) and correct CN prediction (right) text output generated by GPT-3.5 (top), GPT-4 (middle) and Bard (bottom), for two AD subjects and two CN subjects.

Figure 4

MMSE Score vs. prediction classes (AD/CN/Unsure) of Q2 for AD (red) and CN (blue) subjects across three LLM chatbots (circle: GPT-3.5; square: GPT-4; diamond: Bard); the leftmost filled circles depict MMSE score distribution for all AD and CN subjects (true class). The size of the symbols indicates the relative frequency of occurrence of that MMSE value (legend, bottom right).