Reporting guideline for chatbot health advice studies: The CHART statement

CHART Collaborative; Bright Huo¹, Gary Collins², David Chartash³, Arun Thirunavukarasu⁴, Annette Flanagin⁵, Alfonso Iorio⁶, Giovanni Cacciamani⁷, Xi Chen⁸, Nan Liu⁹, Piyush Mathur¹⁰, An-Wen Chan¹¹, Christine Laine¹², Daniela Pacella¹³, Michael Berkwits¹⁴, Stavros A Antoniou¹⁵, Jennifer C Camaradou¹⁶, Carolyn Canfield¹⁷, Michael Mittelman¹⁸, Timothy Feeney¹⁹, Elizabeth Loder²⁰, Riaz Agha²¹, Ashirbani Saha²², Julio Mayol²³, Anthony Sunjaya²⁴, Hugh Harvey²⁵, Jeremy Y Ng²⁶, Tyler McKechnie²⁷, Yung Lee²⁸, Nipun Verma²⁹, Gregor Stiglic³⁰, Melissa McCradden³¹, Karim Ramji³², Vanessa Boudreau²⁷, Monica Ortenzi³³, Joerg Meerpohl³⁴, Per Olav Vandvik³⁵, Thomas Agoritsas³⁶, Diana Samuel³⁷, Helen Frankish³⁸, Michael Anderson³⁹, Xiaomei Yao²², Stacy Loeb⁴⁰, Cynthia Lokker⁶, Xiaoxuan Liu⁴¹, Eliseo Guallar⁴², Gordon Guyatt⁴³

Affiliations

¹ Division of General Surgery, Department of Surgery, McMaster University, Hamilton, Canada. Electronic address: brighthuo@dal.ca.
² UK EQUATOR Centre, University of Oxford, Oxford, UK; Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology & Musculoskeletal Sciences, Botnar Research Centre, University of Oxford, Oxford, UK.
³ Department of Biomedical Informatics and Data Science, Yale University School of Medicine, USA.
⁴ Nuffield Department of Clinical Neurosciences, Medical Sciences Division, University of Oxford, Oxford, UK.
⁵ JAMA and JAMA Network, American Medical Association, USA.
⁶ Department of Health Research Methods, Evidence, and Impact, Department of Medicine, McMaster University, Canada.
⁷ USC Institute of Urology and Catherine and Joseph Aresty, Department of Urology, Keck School of Medicine, University of Southern California, Los Angeles, CA, USA; Artificial Intelligence Center at USC Urology, USC Institute of Urology, University of Southern California, Los Angeles, CA, USA.
⁸ Sports Medicine Center, West China Hospital, Sichuan University, Chengdu, China; Department of Orthopedics and Orthopedic Research Institute, West China Hospital, Sichuan University, Chengdu, China.
⁹ Duke-NUS Medical School, National University of Singapore, Singapore, Singapore.
¹⁰ Cleveland Clinic, Case Western Reserve University, USA.
¹¹ Dept of Medicine, Women's College Research Institute, University of Toronto, Canada.
¹² Annals of Internal Medicine, American College of Physicians, USA; American College of Physicians, USA.
¹³ Department of Public Health, University of Naples Federico II, Italy.
¹⁴ Director, Office of Science Dissemination, Office of Science, Centers for Disease Control and Prevention, Atlanta, GA, USA.
¹⁵ Department of General Surgery, Papageorgiou General Hospital, Thessaloniki, Greece.
¹⁶ British Psychological Society, University of Plymouth, UK.
¹⁷ Innovation Support Unit, Department of Family Practice, University of British Columbia, Canada.
¹⁸ Patient SME, Independent Cybersecurity Professional, USA.
¹⁹ The BMJ, London, UK; Department of Epidemiology, Gillings School of Global Public Health, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA.
²⁰ The BMJ, London, UK; Department of Neurology, Brigham and Women's Hospital, Boston, MA, USA.
²¹ International Journal of Surgery, USA; Eworkflow Ltd, London, UK.
²² Department of Oncology, McMaster University, Canada.
²³ Hospital Clinico San Carlos, Instituto de Investigación Sanitaria San Carlos, Facultad de Medicina Universidad Complutense de Madrid, Spain.
²⁴ The George Institute for Global Health, Tyree Institute of Health Engineering, UNSW Engineering, School of Population Health, UNSW Medicine and Health, Australia.
²⁵ Hardian Health, UK.
²⁶ Centre for Journalology, Ottawa Hospital Research Institute, Ottawa, Canada.
²⁷ Division of General Surgery, Department of Surgery, McMaster University, Hamilton, Canada.
²⁸ Division of General Surgery, Department of Surgery, McMaster University, Hamilton, Canada; Digestive Diseases Institute, Cleveland Clinic, Cleveland, OH, USA.
²⁹ Postgraduate Institute of Medical Education and Research, Chandigarh, India.
³⁰ University of Maribor, Slovenia.
³¹ Australian Institute for Machine Learning (AIML), Australia.
³² Phelix AI, Canada.
³³ Università Politecnica delle Marche, Clinica di Chirurgia Generale e d'Urgenza, Italy.
³⁴ Institute for Evidence in Medicine, Medical Center & Faculty of Medicine, University of Freiburg, Germany; Cochrane Germany, Cochrane Germany Foundation, Freiburg, Germany.
³⁵ Cochrane Germany, Cochrane Germany Foundation, Freiburg, Germany; MAGIC Evidence Ecosystem Foundation, Norway.
³⁶ Department of Health Research Methods, Evidence, and Impact, Department of Medicine, McMaster University, Canada; MAGIC Evidence Ecosystem Foundation, Norway; University Hospitals of Geneva, Switzerland.
³⁷ The Lancet Digital Health, UK.
³⁸ The Lancet, UK.
³⁹ NIHR Clinical Lecturer, Health Organisation, Policy, Economics (HOPE), Centre for Primary Care & Health Services Research, The University of Manchester, UK; Senior Visiting Fellow, LSE Health, London School of Economics and Political Science, UK.
⁴⁰ New York University, Langone Health, USA.
⁴¹ College of Medicine and Health, University of Birmingham, UK.
⁴² School of Global Public Health, New York University, USA.
⁴³ Department of Health Research Methods, Evidence, and Impact, Department of Medicine, McMaster University, Canada; MAGIC Evidence Ecosystem Foundation, Norway.

PMID: 40753040
DOI: 10.1016/j.artmed.2025.103222

Review

Reporting guideline for chatbot health advice studies: The CHART statement

CHART Collaborative et al. Artif Intell Med. 2025 Oct.

. 2025 Oct:168:103222.

doi: 10.1016/j.artmed.2025.103222. Epub 2025 Aug 1.

Authors

Affiliations

¹ Division of General Surgery, Department of Surgery, McMaster University, Hamilton, Canada. Electronic address: brighthuo@dal.ca.
² UK EQUATOR Centre, University of Oxford, Oxford, UK; Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology & Musculoskeletal Sciences, Botnar Research Centre, University of Oxford, Oxford, UK.
³ Department of Biomedical Informatics and Data Science, Yale University School of Medicine, USA.
⁴ Nuffield Department of Clinical Neurosciences, Medical Sciences Division, University of Oxford, Oxford, UK.
⁵ JAMA and JAMA Network, American Medical Association, USA.
⁶ Department of Health Research Methods, Evidence, and Impact, Department of Medicine, McMaster University, Canada.
⁷ USC Institute of Urology and Catherine and Joseph Aresty, Department of Urology, Keck School of Medicine, University of Southern California, Los Angeles, CA, USA; Artificial Intelligence Center at USC Urology, USC Institute of Urology, University of Southern California, Los Angeles, CA, USA.
⁸ Sports Medicine Center, West China Hospital, Sichuan University, Chengdu, China; Department of Orthopedics and Orthopedic Research Institute, West China Hospital, Sichuan University, Chengdu, China.
⁹ Duke-NUS Medical School, National University of Singapore, Singapore, Singapore.
¹⁰ Cleveland Clinic, Case Western Reserve University, USA.
¹¹ Dept of Medicine, Women's College Research Institute, University of Toronto, Canada.
¹² Annals of Internal Medicine, American College of Physicians, USA; American College of Physicians, USA.
¹³ Department of Public Health, University of Naples Federico II, Italy.
¹⁴ Director, Office of Science Dissemination, Office of Science, Centers for Disease Control and Prevention, Atlanta, GA, USA.
¹⁵ Department of General Surgery, Papageorgiou General Hospital, Thessaloniki, Greece.
¹⁶ British Psychological Society, University of Plymouth, UK.
¹⁷ Innovation Support Unit, Department of Family Practice, University of British Columbia, Canada.
¹⁸ Patient SME, Independent Cybersecurity Professional, USA.
¹⁹ The BMJ, London, UK; Department of Epidemiology, Gillings School of Global Public Health, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA.
²⁰ The BMJ, London, UK; Department of Neurology, Brigham and Women's Hospital, Boston, MA, USA.
²¹ International Journal of Surgery, USA; Eworkflow Ltd, London, UK.
²² Department of Oncology, McMaster University, Canada.
²³ Hospital Clinico San Carlos, Instituto de Investigación Sanitaria San Carlos, Facultad de Medicina Universidad Complutense de Madrid, Spain.
²⁴ The George Institute for Global Health, Tyree Institute of Health Engineering, UNSW Engineering, School of Population Health, UNSW Medicine and Health, Australia.
²⁵ Hardian Health, UK.
²⁶ Centre for Journalology, Ottawa Hospital Research Institute, Ottawa, Canada.
²⁷ Division of General Surgery, Department of Surgery, McMaster University, Hamilton, Canada.
²⁸ Division of General Surgery, Department of Surgery, McMaster University, Hamilton, Canada; Digestive Diseases Institute, Cleveland Clinic, Cleveland, OH, USA.
²⁹ Postgraduate Institute of Medical Education and Research, Chandigarh, India.
³⁰ University of Maribor, Slovenia.
³¹ Australian Institute for Machine Learning (AIML), Australia.
³² Phelix AI, Canada.
³³ Università Politecnica delle Marche, Clinica di Chirurgia Generale e d'Urgenza, Italy.
³⁴ Institute for Evidence in Medicine, Medical Center & Faculty of Medicine, University of Freiburg, Germany; Cochrane Germany, Cochrane Germany Foundation, Freiburg, Germany.
³⁵ Cochrane Germany, Cochrane Germany Foundation, Freiburg, Germany; MAGIC Evidence Ecosystem Foundation, Norway.
³⁶ Department of Health Research Methods, Evidence, and Impact, Department of Medicine, McMaster University, Canada; MAGIC Evidence Ecosystem Foundation, Norway; University Hospitals of Geneva, Switzerland.
³⁷ The Lancet Digital Health, UK.
³⁸ The Lancet, UK.
³⁹ NIHR Clinical Lecturer, Health Organisation, Policy, Economics (HOPE), Centre for Primary Care & Health Services Research, The University of Manchester, UK; Senior Visiting Fellow, LSE Health, London School of Economics and Political Science, UK.
⁴⁰ New York University, Langone Health, USA.
⁴¹ College of Medicine and Health, University of Birmingham, UK.
⁴² School of Global Public Health, New York University, USA.
⁴³ Department of Health Research Methods, Evidence, and Impact, Department of Medicine, McMaster University, Canada; MAGIC Evidence Ecosystem Foundation, Norway.

PMID: 40753040
DOI: 10.1016/j.artmed.2025.103222

Abstract

The Chatbot Assessment Reporting Tool (CHART) is a reporting guideline developed to provide reporting recommendations for studies evaluating the performance of generative artificial intelligence (AI)-driven chatbots when summarizing clinical evidence and providing health advice, referred to as Chatbot Health Advice (CHA) studies. CHART was developed in several phases after performing a comprehensive systematic review to identify variation in the conduct, reporting and methodology in CHA studies. Findings from the review were used to develop a draft checklist that was revised through an international, multidisciplinary modified asynchronous Delphi consensus process of 531 stakeholders, three synchronous panel consensus meetings of 48 stakeholders, and subsequent pilot testing of the checklist. CHART includes 12 items and 39 subitems to promote transparent and comprehensive reporting of CHA studies. These include Title (subitem 1a), Abstract/Summary (subitem 1b), Background (subitems 2ab), Model Identifiers (subitem 3ab), Model Details (subitems 4abc), Prompt Engineering (subitems 5ab), Query Strategy (subitems 6abcd), Performance Evaluation (subitems 7ab), Sample Size (subitem 8), Data Analysis (subitem 9a), Results (subitems 10abc), Discussion (subitems 11abc), Disclosures (subitem 12a), Funding (subitem 12b), Ethics (subitem 12c), Protocol (subitem 12d), and Data Availability (subitem 12e). The CHART checklist and corresponding methodological diagram were designed to support key stakeholders including clinicians, researchers, editors, peer reviewers, and readers in reporting, understanding, and interpreting the findings of CHA studies.

Keywords: Generative AI; LLMs; Reporting standards.

PubMed Disclaimer

Conflict of interest statement

Declaration of competing interest All authors have completed the ICMJE uniform disclosure form at www.icmje.org/disclosure-of-interest/and declare: GSC is a National Institute for Health and Care Research (NIHR) Senior Investigator. The views expressed in this article are those of the author(s) and not necessarily those of the NIHR, or the Department of Health and Social Care; AJT has received funding from HealthSense to investigate evidence-based medicine applications of large language models. PM is the co-founder of BrainX LLC; AS has received research funding from the Australian government and is co-founder of BantingMed Pty Ltd.; DS is the Acting Deputy Editor for the Lancet Digital Health; MM has received research funding from The Hospital Research Founding Group; TF sits on the executive committee of MDEpiNet; HF is a Senior Executive Editor for The Lancet; CL is the Editor in Chief of Annals of Internal Medicine; AF is Executive Managing Editor and Vice President, Editorial Operations, JAMA and The JAMA Network; TF and EL are journal editors for the BMJ; RA is the Editor in Chief of International Journal of Surgery; GS is an Executive Editor of Artificial Intelligence in Medicine; SL is a paid consultant for Astellas; DP has received research funding from the Italian Ministry of University and Research; MO is a paid consultant for Theator; TA, POV, GG are board members of the MAGIC Evidence Ecosystem Foundation (www.magicproject.org), a non-for profit organisation, which conducts research and evidence appraisal and guideline methodology and implementation, and which provides a authoring and publication software (MAGICapp) for evidence summaries, guidelines and decision aids.

Publication types

Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions

LinkOut - more resources

Full Text Sources
- ClinicalKey
- Elsevier Science

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Reporting guideline for chatbot health advice studies: The CHART statement

Affiliations

Reporting guideline for chatbot health advice studies: The CHART statement

Authors

Affiliations

Abstract

Conflict of interest statement

Publication types

MeSH terms

LinkOut - more resources

Full Text Sources