A mixed methods crossover randomized controlled trial exploring the experiences, perceptions, and usability of artificial intelligence (ChatGPT) in health sciences education

Mirella Veras^{1

2}, Joseph-Omer Dyer^{3

4}, Holly Shannon⁵, Bryce J M Bogie⁶, Morgan Ronney⁷, Harmehr Sekhon^{8

9}, Derek Rutherford¹⁰, Paulo Goberlânio Barros Silva¹¹, Dahlia Kairy^{6

12}

Affiliations

¹ Department of Physical Therapy, College of Rehabilitation Sciences, University of Manitoba, Winnipeg, MB, Canada.
² Department of Health Sciences, Carleton University, Ottawa, ON, Canada.
³ École de Réadaptation, Faculté de Médecine, Université de Montréal, Montréal, QC, Canada.
⁴ Groupe Interdisciplinaire de Recherche sur la Cognition et le Raisonnement Professionnel, Faculty of Medicine, Université de Montréal, Montréal, QC, Canada.
⁵ Department of Neuroscience, Carleton University, Ottawa, ON, Canada.
⁶ Department of Cellular and Molecular Medicine, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada.
⁷ Teaching and Learning Services, Carleton University, Ottawa, ON, Canada.
⁸ Department of Medicine, Division of Geriatric Medicine, McGill University, and St Mary's Research Centre, Montréal, QC, Canada.
⁹ The Centre for Addiction and Mental Health, Toronto, ON, Canada.
¹⁰ School of Physiotherapy, Dalhousie University, Halifax, NS, Canada.
¹¹ Centro Universitário Christus, Fortaleza, Ceará, Brazil.
¹² Institut Universitaire sur la Réadaptation en Déficience Physique de Montréal, Centre Intégré Universitaire de Santé et Services Sociaux du Centre-Sud-de-l'Île-de-Montréal, Montréal, QC, Canada.

PMID: 39917060
PMCID: PMC11800248
DOI: 10.1177/20552076241298485

A mixed methods crossover randomized controlled trial exploring the experiences, perceptions, and usability of artificial intelligence (ChatGPT) in health sciences education

Mirella Veras et al. Digit Health. 2024.

. 2024 Dec 5:10:20552076241298485.

doi: 10.1177/20552076241298485. eCollection 2024 Jan-Dec.

Authors

Affiliations

¹ Department of Physical Therapy, College of Rehabilitation Sciences, University of Manitoba, Winnipeg, MB, Canada.
² Department of Health Sciences, Carleton University, Ottawa, ON, Canada.
³ École de Réadaptation, Faculté de Médecine, Université de Montréal, Montréal, QC, Canada.
⁴ Groupe Interdisciplinaire de Recherche sur la Cognition et le Raisonnement Professionnel, Faculty of Medicine, Université de Montréal, Montréal, QC, Canada.
⁵ Department of Neuroscience, Carleton University, Ottawa, ON, Canada.
⁶ Department of Cellular and Molecular Medicine, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada.
⁷ Teaching and Learning Services, Carleton University, Ottawa, ON, Canada.
⁸ Department of Medicine, Division of Geriatric Medicine, McGill University, and St Mary's Research Centre, Montréal, QC, Canada.
⁹ The Centre for Addiction and Mental Health, Toronto, ON, Canada.
¹⁰ School of Physiotherapy, Dalhousie University, Halifax, NS, Canada.
¹¹ Centro Universitário Christus, Fortaleza, Ceará, Brazil.
¹² Institut Universitaire sur la Réadaptation en Déficience Physique de Montréal, Centre Intégré Universitaire de Santé et Services Sociaux du Centre-Sud-de-l'Île-de-Montréal, Montréal, QC, Canada.

PMID: 39917060
PMCID: PMC11800248
DOI: 10.1177/20552076241298485

Abstract

Background: Generative artificial intelligence (AI) integrated programs such as Chat Generative Pre-trained Transformers (ChatGPT) are becoming more widespread in educational settings, with mounting ethical and reliability concerns regarding its usage. This paper explores the experiences, perceptions, and usability of ChatGPT in undergraduate health sciences students.

Methods: Twenty-seven students at Carleton University (Canada) were enrolled in a crossover randomized controlled trial study from a Health Sciences course during the Fall 2023 academic term. The intervention condition involved the use of ChatGPT-3.5, whereas the control condition involved using conventional web-based tools. Technology usability was compared between ChatGPT-3.5 and the traditional tools using questionnaires. Focus group discussions were conducted with seven students to further elaborate on student perceptions and experiences. Reflexive thematic analysis was employed to identify themes from the focus group data.

Results: Easiness of learnability for personal use and a perception of quick learnability towards ChatGPT-3.5 were significantly higher, compared to conventional online tools from the Systems Usability Scale. Qualitative results highlighted strong benefits of ChatGPT-3.5, such as being a tool for increased overall productivity and brainstorming. However, students identified challenges associated with reliability and accuracy, and concerns about academic integrity.

Conclusions: Despite the benefits and positive usability of ChatGPT-3.5 identified by students, an explicit need for the development of policies, procedures and regulations remains. An established framework of best practices for the usage of AI within health science education is necessary. This will ensure accountability of users and lead to a more effective integration of AI technologies into academic settings.

Keywords: AI; ChatGPT; artificial intelligence; crossover RCT; education; health sciences; learning outcomes; perceptions; randomized controlled trial; usability.

PubMed Disclaimer

Conflict of interest statement

The data sets used and analyzed during this study are available from the corresponding author upon reasonable request. The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Figures

**Figure 1.**
Schematic summary of the findings from the qualitative analysis. Themes are depicted in boxes and subthemes in circles of corresponding color

See this image and copyright information in PMC

References

1. Deng J, Lin Y. The benefits and challenges of ChatGPT: an overview. Front Comput Intell Syst 2022; 2: 81–83.
1. Hosseini M, Gao CA, Liebovitz DM, et al. An exploratory survey about using ChatGPT in education, healthcare, and research. PLoS One 2023; 18: e0292216. - PMC - PubMed
1. Sallam M. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare 2023; 11: 887. - PMC - PubMed
1. Sallam M, Salim NA, Barakat M, et al. Assessing health Students’ attitudes and usage of ChatGPT in Jordan: validation study. JMIR Med Educ 2023; 9: e48254. - PMC - PubMed
1. Van Dis EAM, Bollen J, Zuidema W, et al. ChatGPT: five priorities for research. Nature 2023; 614: 224–226. - PubMed

LinkOut - more resources

Full Text Sources
- Atypon
- PubMed Central

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

A mixed methods crossover randomized controlled trial exploring the experiences, perceptions, and usability of artificial intelligence (ChatGPT) in health sciences education

Affiliations

A mixed methods crossover randomized controlled trial exploring the experiences, perceptions, and usability of artificial intelligence (ChatGPT) in health sciences education

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

References

LinkOut - more resources

Full Text Sources