The ChatGPT Fact-Check: exploiting the limitations of generative AI to develop evidence-based reasoning skills in college science courses

Abstract

Generative large language models (LLMs) like ChatGPT can quickly produce informative essays on various topics. However, the information generated cannot be fully trusted, as artificial intelligence (AI) can make factual mistakes. This poses challenges for using such tools in college classrooms. To address this, an adaptable assignment called the ChatGPT Fact-Check was developed to teach students in college science courses the benefits of using LLMs for topic exploration while emphasizing the importance of validating their claims based on evidence. The assignment requires students to use ChatGPT to generate essays, evaluate AI-generated sources, and assess the validity of AI-generated scientific claims (based on experimental evidence in primary sources). The assignment reinforces student learning around responsible AI use for exploration while maintaining evidence-based skepticism. The assignment meets objectives around efficiently leveraging beneficial features of AI, distinguishing evidence types, and evidence-based claim evaluation. Its adaptable nature allows integration across diverse courses to teach students to responsibly use AI for learning while maintaining a critical stance.NEW & NOTEWORTHY Generative large language models (LLMs) (e.g., ChatGPT) often produce erroneous information unsupported by scientific evidence. This article outlines how these limitations may be leveraged to develop critical thinking and teach students the importance of evaluating claims based on experimental evidence. Additionally, the activity highlights positive aspects of generative AI to efficiently explore new topics of interest, while maintaining skepticism.

Keywords: artificial intelligence; critical thinking; evidence-based reasoning; source evaluation; topic exploration.