Comparative Performance of Medical Students, ChatGPT-3.5 and ChatGPT-4.0 in Answering Questions From a Brazilian National Medical Exam: Cross-Sectional Questionnaire Study

Abstract

Background: Artificial intelligence has advanced significantly in various fields, including medicine, where tools like ChatGPT (GPT) have demonstrated remarkable capabilities in interpreting and synthesizing complex medical data. Since its launch in 2019, GPT has evolved, with version 4.0 offering enhanced processing power, image interpretation, and more accurate responses. In medicine, GPT has been used for diagnosis, research, and education, achieving significant milestones like passing the United States Medical Licensing Examination. Recent studies show that GPT 4.0 outperforms earlier versions and even medical students on medical exams.

Objective: This study aimed to evaluate and compare the performance of GPT versions 3.5 and 4.0 on Brazilian Progress Tests (PT) from 2021 to 2023, analyzing their accuracy compared to medical students.

Methods: A cross-sectional observational study was conducted using 333 multiple-choice questions from the PT, excluding questions with images and those nullified or repeated. All questions were presented sequentially without modification to their structure. The performance of GPT versions was compared using statistical methods and medical students' scores were included for context.

Results: There was a statistically significant difference in total performance scores across the 2021, 2022, and 2023 exams between GPT-3.5 and GPT-4.0 (P=.03). However, this significance did not remain after Bonferroni correction. On average, GPT v3.5 scored 68.4%, whereas v4.0 achieved 87.2%, reflecting an absolute improvement of 18.8% and a relative increase of 27.4% in accuracy. When broken down by subject, the average scores for GPT-3.5 and GPT-4.0, respectively, were as follows: surgery (73.5% vs 88.0%, P=.03), basic sciences (77.5% vs 96.2%, P=.004), internal medicine (61.5% vs 75.1%, P=.14), gynecology and obstetrics (64.5% vs 94.8%, P=.002), pediatrics (58.5% vs 80.0%, P=.02), and public health (77.8% vs 89.6%, P=.02). After Bonferroni correction, only basic sciences and gynecology and obstetrics retained statistically significant differences.

Conclusions: GPT-4.0 demonstrates superior accuracy compared to its predecessor in answering medical questions on the PT. These results are similar to other studies, indicating that we are approaching a new revolution in medicine.

Keywords: AI; ChatGPT; academic performance; accuracy; artificial intelligence; biomedical technology; ethics; exam questions; intelligent systems; medical data; medical education; medical ethics; medical exam; medical school; medical student; observational study.

Figure 1.. Comparison of performance accuracy of GPT-3.5, GPT-4.0, and medical students’ score by year.

Figure 2.. Comparison of performance accuracy of GPT-3.5, GPT-4.0, and sixth year medical students’ score by year.

Figure 3.. Comparison of overall accuracy scores between GPT-3.5, GPT 4.0, and medical students’ score in the 2021 progress test.

Figure 4.. Comparison of overall accuracy scores between GPT-3.5, GPT-4.0, and medical students’ score in the 2022 progress test.

Figure 5.. Comparison of overall accuracy scores between GPT-3.5, GPT-4.0, and medical student’s score in the 2023 progress test.

Figure 6.. Comparison of GPT-3.5, GPT-4.0, and medical students’ combined scores in the 2022-2023 progress test.

Figure 7.. Comparison of GPT-3.5, GPT-4.0, and sixth-year medical students’ average subject accuracy score of the 2021-2023 progress test.

Figure 8.. Comparison of GPT-3.5, GPT-4.0, and sixth-year medical student’s radar score in the 2022-2023 progress test.