Reporting guideline for Chatbot Health Advice studies: the CHART statement

Bright Huo¹, Gary Collins^{2

3}, David Chartash⁴, Arun Thirunavukarasu⁵, Annette Flanagin⁶, Alfonso Iorio⁷, Giovanni Cacciamani^{8

9}, Xi Chen^{10

11}, Nan Liu¹², Piyush Mathur¹³, An-Wen Chan¹⁴, Christine Laine^{15

16}, Daniela Pacella¹⁷, Michael Berkwits¹⁸, Stavros A Antoniou¹⁹, Jennifer C Camaradou²⁰, Carolyn Canfield²¹, Michael Mittelman²², Timothy Feeney^{23

24}, Elizabeth Loder^{23

25}, Riaz Agha^{26

27}, Ashirbani Saha²⁸, Julio Mayol²⁹, Anthony Sunjaya³⁰, Hugh Harvey³¹, Jeremy Y Ng³², Tyler McKechnie³³, Yung Lee^{33

34}, Nipun Verma³⁵, Gregor Stiglic³⁶, Melissa McCradden³⁷, Karim Ramji³⁸, Vanessa Boudreau³³, Monica Ortenzi³⁹, Joerg Meerpohl^{40

41}, Per Olav Vandvik^{41

42}, Thomas Agoritsas^{7

42

43}, Diana Samuel⁴⁴, Helen Frankish⁴⁵, Michael Anderson^{46

47}, Xiaomei Yao²⁸, Stacy Loeb⁴⁸, Cynthia Lokker⁷, Xiaoxuan Liu⁴⁹, Eliseo Guallar⁵⁰, Gordon Guyatt^{7

42}; CHART Collaborative

Affiliations

¹ Division of General Surgery, Department of Surgery, McMaster University, Hamilton, Canada. brighthuo@dal.ca.
² UK EQUATOR Centre, University of Oxford, Oxford, UK.
³ Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology & 401 Musculoskeletal Sciences, Botnar Research Centre,, University of Oxford, Oxford, UK.
⁴ Department of Biomedical Informatics and Data Science, Yale University School of Medicine, New Haven, USA.
⁵ Nuffield Department of Clinical Neurosciences, Medical Sciences Division, University of Oxford, Oxford, UK.
⁶ JAMA and JAMA Network, American Medical Association, Chicago, USA.
⁷ Department of Health Research Methods, Evidence, and Impact; Department of Medicine, McMaster University, Hamilton, Canada.
⁸ USC Institute of Urology and Catherine and Joseph Aresty Department of Urology, Keck School of Medicine, University of Southern California, Los Angeles, CA, USA.
⁹ Artificial Intelligence Center at USC Urology, USC Institute of Urology, University of Southern California, Los Angeles, CA, USA.
¹⁰ Sports Medicine Center, West China Hospital, Sichuan University, Chengdu, China.
¹¹ Department of Orthopedics and Orthopedic Research Institute, West China Hospital, Sichuan University, Chengdu, China.
¹² Duke-NUS Medical School, National University of Singapore, Singapore, Singapore.
¹³ Cleveland Clinic, Case Western Reserve University, Cleveland, USA.
¹⁴ Department of Medicine, Women's College Research Institute, University of Toronto, Toronto, Canada.
¹⁵ Annals of Internal Medicine, American College of Physicians, Philadelphia, USA.
¹⁶ American College of Physicians, Philadelphia, USA.
¹⁷ Department of Public Health, University of Naples Federico II, Naples, Italy.
¹⁸ Director, Office of Science Dissemination, Office of Science, Centers for Disease Control and Prevention, Atlanta, GA, USA.
¹⁹ Department of General Surgery, Papageorgiou General Hospital, Thessaloniki, Greece.
²⁰ British Psychological Society, University of Plymouth, Plymouth, UK.
²¹ Innovation Support Unit, Department of Family Practice, University of British Columbia, Vancouver, Canada.
²² Patient SME, Independent Cybersecurity Professional, London, UK.
²³ The BMJ, London, UK.
²⁴ Department of Epidemiology, Gillings School of Global Public Health, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA.
²⁵ Department of Neurology, Brigham and Women's Hospital, Boston, MA, USA.
²⁶ International Journal of Surgery, London, UK.
²⁷ Eworkflow Ltd, London, UK.
²⁸ Department of Oncology, McMaster University, Hamilton, Canada.
²⁹ Hospital Clinico San Carlos, Instituto de Investigación Sanitaria San Carlos, Facultad de Medicina Universidad Complutense de Madrid, Madrid, Spain.
³⁰ The George Institute for Global Health; Tyree Institute of Health Engineering, UNSW Engineering; School of Population Health, UNSW Medicine and Health, Sydney, Australia.
³¹ Hardian Health, London, UK.
³² Centre for Journalology, Ottawa Hospital Research Institute, Ottawa, Canada.
³³ Division of General Surgery, Department of Surgery, McMaster University, Hamilton, Canada.
³⁴ Digestive Diseases Institute, Cleveland Clinic, Cleveland, OH, USA.
³⁵ Postgraduate Institute of Medical Education and Research, Chandigarh, India.
³⁶ University of Maribor, Maribor, Slovenia.
³⁷ Australian Institute for Machine Learning (AIML), Adelaide, Australia.
³⁸ Phelix AI, Toronto, Canada.
³⁹ Università Politecnica delle Marche, Clinica di Chirurgia Generale e d'Urgenza, Ancona, Italy.
⁴⁰ Institute for Evidence in Medicine, Medical Center & Faculty of Medicine, University of Freiburg, Freiburg im Breisgau, Germany.
⁴¹ Cochrane Germany, Cochrane Germany Foundation, Freiburg, Germany.
⁴² MAGIC Evidence Ecosystem Foundation, Oslo, Norway.
⁴³ University Hospitals of Geneva, Geneva, Switzerland.
⁴⁴ The Lancet Digital Health, London, UK.
⁴⁵ The Lancet, London, UK.
⁴⁶ NIHR Clinical Lecturer, Health Organisation, Policy, Economics (HOPE), Centre for Primary Care & Health Services Research, The University of Manchester, Manchester, UK.
⁴⁷ Senior Visiting Fellow, LSE Health, London School of Economics and Political Science, Manchester, UK.
⁴⁸ New York University Langone Health, New York City, USA.
⁴⁹ College of Medicine and Health, University of Birmingham, Birmingham, UK.
⁵⁰ School of Global Public Health, New York University, New York City, USA.

PMID: 40745595
PMCID: PMC12315282
DOI: 10.1186/s12916-025-04274-w

Review

Reporting guideline for Chatbot Health Advice studies: the CHART statement

Bright Huo et al. BMC Med. 2025.

. 2025 Aug 1;23(1):447.

doi: 10.1186/s12916-025-04274-w.

Authors

Affiliations

¹ Division of General Surgery, Department of Surgery, McMaster University, Hamilton, Canada. brighthuo@dal.ca.
² UK EQUATOR Centre, University of Oxford, Oxford, UK.
³ Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology & 401 Musculoskeletal Sciences, Botnar Research Centre,, University of Oxford, Oxford, UK.
⁴ Department of Biomedical Informatics and Data Science, Yale University School of Medicine, New Haven, USA.
⁵ Nuffield Department of Clinical Neurosciences, Medical Sciences Division, University of Oxford, Oxford, UK.
⁶ JAMA and JAMA Network, American Medical Association, Chicago, USA.
⁷ Department of Health Research Methods, Evidence, and Impact; Department of Medicine, McMaster University, Hamilton, Canada.
⁸ USC Institute of Urology and Catherine and Joseph Aresty Department of Urology, Keck School of Medicine, University of Southern California, Los Angeles, CA, USA.
⁹ Artificial Intelligence Center at USC Urology, USC Institute of Urology, University of Southern California, Los Angeles, CA, USA.
¹⁰ Sports Medicine Center, West China Hospital, Sichuan University, Chengdu, China.
¹¹ Department of Orthopedics and Orthopedic Research Institute, West China Hospital, Sichuan University, Chengdu, China.
¹² Duke-NUS Medical School, National University of Singapore, Singapore, Singapore.
¹³ Cleveland Clinic, Case Western Reserve University, Cleveland, USA.
¹⁴ Department of Medicine, Women's College Research Institute, University of Toronto, Toronto, Canada.
¹⁵ Annals of Internal Medicine, American College of Physicians, Philadelphia, USA.
¹⁶ American College of Physicians, Philadelphia, USA.
¹⁷ Department of Public Health, University of Naples Federico II, Naples, Italy.
¹⁸ Director, Office of Science Dissemination, Office of Science, Centers for Disease Control and Prevention, Atlanta, GA, USA.
¹⁹ Department of General Surgery, Papageorgiou General Hospital, Thessaloniki, Greece.
²⁰ British Psychological Society, University of Plymouth, Plymouth, UK.
²¹ Innovation Support Unit, Department of Family Practice, University of British Columbia, Vancouver, Canada.
²² Patient SME, Independent Cybersecurity Professional, London, UK.
²³ The BMJ, London, UK.
²⁴ Department of Epidemiology, Gillings School of Global Public Health, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA.
²⁵ Department of Neurology, Brigham and Women's Hospital, Boston, MA, USA.
²⁶ International Journal of Surgery, London, UK.
²⁷ Eworkflow Ltd, London, UK.
²⁸ Department of Oncology, McMaster University, Hamilton, Canada.
²⁹ Hospital Clinico San Carlos, Instituto de Investigación Sanitaria San Carlos, Facultad de Medicina Universidad Complutense de Madrid, Madrid, Spain.
³⁰ The George Institute for Global Health; Tyree Institute of Health Engineering, UNSW Engineering; School of Population Health, UNSW Medicine and Health, Sydney, Australia.
³¹ Hardian Health, London, UK.
³² Centre for Journalology, Ottawa Hospital Research Institute, Ottawa, Canada.
³³ Division of General Surgery, Department of Surgery, McMaster University, Hamilton, Canada.
³⁴ Digestive Diseases Institute, Cleveland Clinic, Cleveland, OH, USA.
³⁵ Postgraduate Institute of Medical Education and Research, Chandigarh, India.
³⁶ University of Maribor, Maribor, Slovenia.
³⁷ Australian Institute for Machine Learning (AIML), Adelaide, Australia.
³⁸ Phelix AI, Toronto, Canada.
³⁹ Università Politecnica delle Marche, Clinica di Chirurgia Generale e d'Urgenza, Ancona, Italy.
⁴⁰ Institute for Evidence in Medicine, Medical Center & Faculty of Medicine, University of Freiburg, Freiburg im Breisgau, Germany.
⁴¹ Cochrane Germany, Cochrane Germany Foundation, Freiburg, Germany.
⁴² MAGIC Evidence Ecosystem Foundation, Oslo, Norway.
⁴³ University Hospitals of Geneva, Geneva, Switzerland.
⁴⁴ The Lancet Digital Health, London, UK.
⁴⁵ The Lancet, London, UK.
⁴⁶ NIHR Clinical Lecturer, Health Organisation, Policy, Economics (HOPE), Centre for Primary Care & Health Services Research, The University of Manchester, Manchester, UK.
⁴⁷ Senior Visiting Fellow, LSE Health, London School of Economics and Political Science, Manchester, UK.
⁴⁸ New York University Langone Health, New York City, USA.
⁴⁹ College of Medicine and Health, University of Birmingham, Birmingham, UK.
⁵⁰ School of Global Public Health, New York University, New York City, USA.

PMID: 40745595
PMCID: PMC12315282
DOI: 10.1186/s12916-025-04274-w

Abstract

Background: The Chatbot Assessment Reporting Tool (CHART) is a reporting guideline developed to provide reporting recommendations for studies evaluating the performance of generative artificial intelligence (AI)-driven chatbots when summarizing clinical evidence and providing health advice, referred to as Chatbot Health Advice (CHA) studies.

Methods: CHART was developed in several phases after performing a comprehensive systematic review to identify variation in the conduct, reporting, and methodology in CHA studies. Findings from the review were used to develop a draft checklist that was revised through an international, multidisciplinary modified asynchronous Delphi consensus process of 531 stakeholders, three synchronous panel consensus meetings of 48 stakeholders, and subsequent pilot testing of the checklist.

Results: CHART includes 12 items and 39 subitems to promote transparent and comprehensive reporting of CHA studies. These include Title (subitem 1a), Abstract/Summary (subitem 1b), Background (subitems 2ab), Model Identifiers (subitems 3ab), Model Details (subitems 4abc), Prompt Engineering (subitems 5ab), Query Strategy (subitems 6abcd), Performance Evaluation (subitems 7ab), Sample Size (subitem 8), Data Analysis (subitem 9a), Results (subitems 10abc), Discussion (subitems 11abc), Disclosures (subitem 12a), Funding (subitem 12b), Ethics (subitem 12c), Protocol (subitem 12d), and Data Availability (subitem 12e).

Conclusion: The CHART checklist and corresponding methodological diagram were designed to support key stakeholders including clinicians, researchers, editors, peer reviewers, and readers in reporting, understanding, and interpreting the findings of CHA studies.

Keywords: Generative AI; LLMs; Reporting standards.

PubMed Disclaimer

Conflict of interest statement

Declarations. Ethics approval and consent to participate: Ethics approval was submitted to and waived by the Hamilton Integrated Research Ethics Board (HiREB #17025). Consent for publication: Not applicable. Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/disclosure-of-interest/ and declare: GSC is a National Institute for Health and Care Research (NIHR) Senior Investigator. The views expressed in this article are those of the author(s) and not necessarily those of the NIHR, or the Department of Health and Social Care; AJT has received funding from HealthSense to investigate evidence-based medicine applications of large language models. PM is the co-founder of BrainX LLC; AS has received research funding from the Australian government and is co-founder of BantingMed Pty Ltd; DS is the Acting Deputy Editor for the Lancet Digital Health; MM has received research funding from The Hospital Research Founding Group; TF sits on the executive committee of MDEpiNet; HF is a Senior Executive Editor for The Lancet; CL is the Editor in Chief of Annals of Internal Medicine; AF is Executive Managing Editor and Vice President, Editorial Operations, JAMA and The JAMA Network; TF and EL are journal editors for the BMJ; RA is the Editor in Chief of International Journal of Surgery; GS is an Executive Editor of Artificial Intelligence in Medicine; SL is a paid consultant for Astellas; DP has received research funding from the Italian Ministry of University and Research; MO is a paid consultant for Theator; TA, POV, GG are board member of the MAGIC Evidence Ecosystem Foundation ( www.magicproject.org ), a non-for profit organization, which conducts research and evidence appraisal and guideline methodology and implementation, and which provides a authoring and publication software (MAGICapp) for evidence summaries, guidelines and decision aids.

Figures

**Fig. 1**
The CHART Methodological Diagram

See this image and copyright information in PMC

References

1. Kolbinger FR, Veldhuizen GP, Zhu J, Truhn D, Kather JN. Reporting guidelines in medical artificial intelligence: a systematic review and meta-analysis. Commun Med. 2024;4:1. - PMC - PubMed
1. Han R, Acosta JN, Shakeri Z, Ioannidis JPA, Topol EJ, Rajpurkar P. Randomised controlled trials evaluating artificial intelligence in clinical practice: a scoping review. Lancet Digit Health. 2024;6:e367–73. - PMC - PubMed
1. Huo B, Cacciamani GE, Collins GS, McKechnie T, Lee Y, Guyatt G. Reporting standards for the use of large language model-linked chatbots for health advice. Nat Med. 2023;29:2988. - PubMed
1. Huo B, McKechnie T, Ortenzi M, Lee Y, Antoniou S, Mayol J, et al. Dr. GPT will see you now: the ability of large language model-linked chatbots to provide colorectal cancer screening recommendations. Health Technol. 2024;14:463–9.
1. Huo B, Marfo N, Sylla P, Calabrese E, Kumar S, Slater BJ, et al. Clinical artificial intelligence: teaching a large language model to generate recommendations that align with guidelines for the surgical management of GERD. Surg Endosc. 2024;38:5668–77. - PubMed

Publication types

Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions

LinkOut - more resources

Full Text Sources
- BioMed Central
- PubMed Central

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Reporting guideline for Chatbot Health Advice studies: the CHART statement

Affiliations

Reporting guideline for Chatbot Health Advice studies: the CHART statement

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

References

Publication types

MeSH terms

LinkOut - more resources

Full Text Sources