. 2022 May 18:377:e070904.

doi: 10.1136/bmj-2022-070904.

Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI

Baptiste Vasey^{1

2

3}, Myura Nagendran⁴, Bruce Campbell^{5

6}, David A Clifton², Gary S Collins⁷, Spiros Denaxas^{8

9

10

11}, Alastair K Denniston^{12

13

14}, Livia Faes¹⁴, Bart Geerts¹⁵, Mudathir Ibrahim^{16

17}, Xiaoxuan Liu^{12

13}, Bilal A Mateen^{8

18

19}, Piyush Mathur²⁰, Melissa D McCradden^{21

22}, Lauren Morgan²³, Johan Ordish²⁴, Campbell Rogers²⁵, Suchi Saria^{26

27}, Daniel S W Ting^{28

29}, Peter Watkinson^{3

30}, Wim Weber³¹, Peter Wheatstone³², Peter McCulloch¹⁶; DECIDE-AI expert group

Collaborators, Affiliations

Collaborators

DECIDE-AI expert group:
Aaron Y Lee, Alan G Fraser, Ali Connell, Alykhan Vira, Andre Esteva, Andrew D Althouse, Andrew L Beam, Anne de Hond, Anne-Laure Boulesteix, Anthony Bradlow, Ari Ercole, Arsenio Paez, Athanasios Tsanas, Barry Kirby, Ben Glocker, Carmelo Velardo, Chang Min Park, Charisma Hehakaya, Chris Baber, Chris Paton, Christian Johner, Christopher J Kelly, Christopher J Vincent, Christopher Yau, Clare McGenity, Constantine Gatsonis, Corinne Faivre-Finn, Crispin Simon, Danielle Sent, Danilo Bzdok, Darren Treanor, David C Wong, David F Steiner, David Higgins, Dawn Benson, Declan P O'Regan, Dinesh V Gunasekaran, Dominic Danks, Emanuele Neri, Evangelia Kyrimi, Falk Schwendicke, Farah Magrabi, Frances Ives, Frank E Rademakers, George E Fowler, Giuseppe Frau, H D Jeffry Hogg, Hani J Marcus, Heang-Ping Chan, Henry Xiang, Hugh F McIntyre, Hugh Harvey, Hyungjin Kim, Ibrahim Habli, James C Fackler, James Shaw, Janet Higham, Jared M Wohlgemut, Jaron Chong, Jean-Emmanuel Bibault, Jérémie F Cohen, Jesper Kers, Jessica Morley, Joachim Krois, Joao Monteiro, Joel Horovitz, John Fletcher, Jonathan Taylor, Jung Hyun Yoon, Karandeep Singh, Karel G M Moons, Kassandra Karpathakis, Ken Catchpole, Kerenza Hood, Konstantinos Balaskas, Konstantinos Kamnitsas, Laura Militello, Laure Wynants, Lauren Oakden-Rayner, Laurence B Lovat, Luc J M Smits, Ludwig C Hinske, M Khair ElZarrad, Maarten van Smeden, Mara Giavina-Bianchi, Mark Daley, Mark P Sendak, Mark Sujan, Maroeska Rovers, Matthew DeCamp, Matthew Woodward, Matthieu Komorowski, Max Marsden, Maxine Mackintosh, Michael D Abramoff, Miguel Ángel Armengol de la Hoz, Neale Hambidge, Neil Daly, Niels Peek, Oliver Redfern, Omer F Ahmad, Patrick M Bossuyt, Pearse A Keane, Pedro N P Ferreira, Petra Schnell-Inderst, Pietro Mascagni, Prokar Dasgupta, Pujun Guan, Rachel Barnett, Rawen Kader, Reena Chopra, Ritse M Mann, Rupa Sarkar, Saana M Mäenpää, Samuel G Finlayson, Sarah Vollam, Sebastian J Vollmer, Seong Ho Park, Shakir Laher, Shalmali Joshi, Siri L van der Meijden, Susan C Shelmerdine, Tien-En Tan, Tom Jw Stocker, Valentina Giannini, Vince I Madai, Virginia Newcombe, Wei Yan Ng, Wendy A Rogers, William Ogallo, Yoonyoung Park, Zane B Perkins

Affiliations

¹ Nuffield Department of Surgical Sciences, University of Oxford, Oxford, UK baptiste.vasey@gmail.com.
² Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK.
³ Critical Care Research Group, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK.
⁴ UKRI Centre for Doctoral Training in AI for Healthcare, Imperial College London, London, UK.
⁵ University of Exeter Medical School, Exeter, UK.
⁶ Royal Devon and Exeter Hospital, Exeter, UK.
⁷ Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK.
⁸ Institute of Health Informatics, University College London, London, UK.
⁹ British Heart Foundation Data Science Centre, London, UK.
¹⁰ Health Data Research UK, London, UK.
¹¹ UCL Hospitals Biomedical Research Centre, London, UK.
¹² University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK.
¹³ Academic Unit of Ophthalmology, Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK.
¹⁴ Moorfields Eye Hospital NHS Foundation Trust, London, UK.
¹⁵ Healthplus.ai-R&D, Amsterdam, Netherlands.
¹⁶ Nuffield Department of Surgical Sciences, University of Oxford, Oxford, UK.
¹⁷ Department of Surgery, Maimonides Medical Center, New York, NY, USA.
¹⁸ Wellcome Trust, London, UK.
¹⁹ Alan Turing Institute, London, UK.
²⁰ Department of General Anesthesiology, Anesthesiology Institute, Cleveland Clinic, Cleveland, OH, USA.
²¹ Hospital for Sick Children, Toronto, ON, Canada.
²² Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada.
²³ Morgan Human Systems, Shrewsbury, UK.
²⁴ The Medicines and Healthcare products Regulatory Agency, London, UK.
²⁵ HeartFlow, Redwood City, CA, USA.
²⁶ Departments of Computer Science, Statistics, and Health Policy, and Division of Informatics, Johns Hopkins University, Baltimore, MD, USA.
²⁷ Bayesian Health, New York, NY, USA.
²⁸ Singapore National Eye Center, Singapore Eye Research Institute, Singapore.
²⁹ Duke-NUS Medical School, National University of Singapore, Singapore.
³⁰ NIHR Biomedical Research Centre Oxford, Oxford University Hospitals NHS Trust, Oxford, UK.
³¹ The BMJ, London, UK.
³² School of Medicine, University of Leeds, Leeds, UK.

PMID: 35584845
PMCID: PMC9116198
DOI: 10.1136/bmj-2022-070904

Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI

Baptiste Vasey et al. BMJ. 2022.

. 2022 May 18:377:e070904.

doi: 10.1136/bmj-2022-070904.

Authors

Collaborators

DECIDE-AI expert group:
Aaron Y Lee, Alan G Fraser, Ali Connell, Alykhan Vira, Andre Esteva, Andrew D Althouse, Andrew L Beam, Anne de Hond, Anne-Laure Boulesteix, Anthony Bradlow, Ari Ercole, Arsenio Paez, Athanasios Tsanas, Barry Kirby, Ben Glocker, Carmelo Velardo, Chang Min Park, Charisma Hehakaya, Chris Baber, Chris Paton, Christian Johner, Christopher J Kelly, Christopher J Vincent, Christopher Yau, Clare McGenity, Constantine Gatsonis, Corinne Faivre-Finn, Crispin Simon, Danielle Sent, Danilo Bzdok, Darren Treanor, David C Wong, David F Steiner, David Higgins, Dawn Benson, Declan P O'Regan, Dinesh V Gunasekaran, Dominic Danks, Emanuele Neri, Evangelia Kyrimi, Falk Schwendicke, Farah Magrabi, Frances Ives, Frank E Rademakers, George E Fowler, Giuseppe Frau, H D Jeffry Hogg, Hani J Marcus, Heang-Ping Chan, Henry Xiang, Hugh F McIntyre, Hugh Harvey, Hyungjin Kim, Ibrahim Habli, James C Fackler, James Shaw, Janet Higham, Jared M Wohlgemut, Jaron Chong, Jean-Emmanuel Bibault, Jérémie F Cohen, Jesper Kers, Jessica Morley, Joachim Krois, Joao Monteiro, Joel Horovitz, John Fletcher, Jonathan Taylor, Jung Hyun Yoon, Karandeep Singh, Karel G M Moons, Kassandra Karpathakis, Ken Catchpole, Kerenza Hood, Konstantinos Balaskas, Konstantinos Kamnitsas, Laura Militello, Laure Wynants, Lauren Oakden-Rayner, Laurence B Lovat, Luc J M Smits, Ludwig C Hinske, M Khair ElZarrad, Maarten van Smeden, Mara Giavina-Bianchi, Mark Daley, Mark P Sendak, Mark Sujan, Maroeska Rovers, Matthew DeCamp, Matthew Woodward, Matthieu Komorowski, Max Marsden, Maxine Mackintosh, Michael D Abramoff, Miguel Ángel Armengol de la Hoz, Neale Hambidge, Neil Daly, Niels Peek, Oliver Redfern, Omer F Ahmad, Patrick M Bossuyt, Pearse A Keane, Pedro N P Ferreira, Petra Schnell-Inderst, Pietro Mascagni, Prokar Dasgupta, Pujun Guan, Rachel Barnett, Rawen Kader, Reena Chopra, Ritse M Mann, Rupa Sarkar, Saana M Mäenpää, Samuel G Finlayson, Sarah Vollam, Sebastian J Vollmer, Seong Ho Park, Shakir Laher, Shalmali Joshi, Siri L van der Meijden, Susan C Shelmerdine, Tien-En Tan, Tom Jw Stocker, Valentina Giannini, Vince I Madai, Virginia Newcombe, Wei Yan Ng, Wendy A Rogers, William Ogallo, Yoonyoung Park, Zane B Perkins

Affiliations

¹ Nuffield Department of Surgical Sciences, University of Oxford, Oxford, UK baptiste.vasey@gmail.com.
² Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK.
³ Critical Care Research Group, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK.
⁴ UKRI Centre for Doctoral Training in AI for Healthcare, Imperial College London, London, UK.
⁵ University of Exeter Medical School, Exeter, UK.
⁶ Royal Devon and Exeter Hospital, Exeter, UK.
⁷ Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK.
⁸ Institute of Health Informatics, University College London, London, UK.
⁹ British Heart Foundation Data Science Centre, London, UK.
¹⁰ Health Data Research UK, London, UK.
¹¹ UCL Hospitals Biomedical Research Centre, London, UK.
¹² University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK.
¹³ Academic Unit of Ophthalmology, Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK.
¹⁴ Moorfields Eye Hospital NHS Foundation Trust, London, UK.
¹⁵ Healthplus.ai-R&D, Amsterdam, Netherlands.
¹⁶ Nuffield Department of Surgical Sciences, University of Oxford, Oxford, UK.
¹⁷ Department of Surgery, Maimonides Medical Center, New York, NY, USA.
¹⁸ Wellcome Trust, London, UK.
¹⁹ Alan Turing Institute, London, UK.
²⁰ Department of General Anesthesiology, Anesthesiology Institute, Cleveland Clinic, Cleveland, OH, USA.
²¹ Hospital for Sick Children, Toronto, ON, Canada.
²² Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada.
²³ Morgan Human Systems, Shrewsbury, UK.
²⁴ The Medicines and Healthcare products Regulatory Agency, London, UK.
²⁵ HeartFlow, Redwood City, CA, USA.
²⁶ Departments of Computer Science, Statistics, and Health Policy, and Division of Informatics, Johns Hopkins University, Baltimore, MD, USA.
²⁷ Bayesian Health, New York, NY, USA.
²⁸ Singapore National Eye Center, Singapore Eye Research Institute, Singapore.
²⁹ Duke-NUS Medical School, National University of Singapore, Singapore.
³⁰ NIHR Biomedical Research Centre Oxford, Oxford University Hospitals NHS Trust, Oxford, UK.
³¹ The BMJ, London, UK.
³² School of Medicine, University of Leeds, Leeds, UK.

PMID: 35584845
PMCID: PMC9116198
DOI: 10.1136/bmj-2022-070904

Abstract

A growing number of artificial intelligence (AI)-based clinical decision support systems are showing promising performance in preclinical, in silico, evaluation, but few have yet demonstrated real benefit to patient care. Early stage clinical evaluation is important to assess an AI system’s actual clinical performance at small scale, ensure its safety, evaluate the human factors surrounding its use, and pave the way to further large scale trials. However, the reporting of these early studies remains inadequate. The present statement provides a multistakeholder, consensus-based reporting guideline for the Developmental and Exploratory Clinical Investigations of DEcision support systems driven by Artificial Intelligence (DECIDE-AI). We conducted a two round, modified Delphi process to collect and analyse expert opinion on the reporting of early clinical evaluation of AI systems. Experts were recruited from 20 predefined stakeholder categories. The final composition and wording of the guideline was determined at a virtual consensus meeting. The checklist and the Explanation & Elaboration (E&E) sections were refined based on feedback from a qualitative evaluation process. 123 experts participated in the first round of Delphi, 138 in the second, 16 in the consensus meeting, and 16 in the qualitative evaluation. The DECIDE-AI reporting guideline comprises 17 AI specific reporting items (made of 28 subitems) and 10 generic reporting items, with an E&E paragraph provided for each. Through consultation and consensus with a range of stakeholders, we have developed a guideline comprising key items that should be reported in early stage clinical studies of AI-based decision support systems in healthcare. By providing an actionable checklist of minimal reporting items, the DECIDE-AI guideline will facilitate the appraisal of these studies and replicability of their findings.

PubMed Disclaimer

Conflict of interest statement

Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: MN consults for Cera Care, a technology enabled homecare provider. BC was a Non-Executive Director of the UK Medicines and Healthcare products Regulatory Agency (MHRA) from September 2015 until 31 August 2021. DC receives consulting fees from Oxford University Innovation, Biobeats, Sensyne Health, and has advisory role with Bristol Myers Squibb. BG has received consultancy and research grants from Philips NV and Edwards Lifesciences, and is owner and board member of Healthplus.ai BV and its subsidiaries. XL has advisory roles with the National Screening Committee UK, the WHO/ITU focus group for AI in health and the AI in Health and Care Award Evaluation Advisory Group (NHSX, AAC). PMa is the cofounder of BrainX and BrainX Community. MM reports consulting fees from AMS Healthcare, and honorariums from the Osgoode Law School and Toronto Pain Institute. LM is director and owner of Morgan Human Systems. JO holds an honorary post as an Associate of Hughes Hall, University of Cambridge. CR is an employee of HeartFlow, including salary and equity. SS has received honorariums from several universities and pharmaceutical companies for talks on digital health and AI. SS has advisory roles in Child Health Imprints, Duality Tech, Halcyon Health, and Bayesian Health. SS is on the board of Bayesian Health. This arrangement has been reviewed and approved by Johns Hopkins in accordance with its conflict-of-interest policies. DSWT holds patents linked to AI driven technologies, and a co-founder and equity holder for EyRIS. PWa declares grants, consulting fees and stocks from Sensyne Health and holds patents linked to AI driven technologies. PMc has advisory role for WEISS International and the technology incubator PhD programme at University College London. BV, GSC, AKD, LF, MI, BAM, SD, PWh, and WW declare no financial relationships with any organisations that might have an interest in the submitted work in the previous three years and no other relationships or activities that could appear to have influenced the submitted work.

Figures

**Fig 1**
Comparison of development pathways for drug therapies, artificial intelligence (AI) in healthcare, and surgical innovation. The coloured lines represent reporting guidelines, some of which are study design specific (TRIPOD-AI, STARD-AI, SPIRIT/CONSORT, SPIRIT/CONSORT-AI), others stage specific (DECIDE-AI, IDEAL). Depending on the context, more than one study design can be appropriate for each stage. *Only apply to AI in healthcare

See this image and copyright information in PMC

References

1. Skivington K, Matthews L, Simpson SA, et al. . A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ 2021;374:n2061. 10.1136/bmj.n2061 - DOI - PMC - PubMed
1. Liu X, Faes L, Kale AU, et al. . A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit Health 2019;1:e271-97. 10.1016/S2589-7500(19)30123-2. - DOI - PubMed
1. Vasey B, Ursprung S, Beddoe B, et al. . Association of Clinician Diagnostic Performance With Machine Learning-Based Decision Support Systems: A Systematic Review. JAMA Netw Open 2021;4:e211276. 10.1001/jamanetworkopen.2021.1276 - DOI - PMC - PubMed
1. Freeman K, Geppert J, Stinton C, et al. . Use of artificial intelligence for image analysis in breast cancer screening programmes: systematic review of test accuracy. BMJ 2021;374:n1872. 10.1136/bmj.n1872 - DOI - PMC - PubMed
1. Keane PA, Topol EJ. With an eye to AI and autonomous diagnosis. NPJ Digit Med 2018;1:40. - PMC - PubMed

MeSH terms

Actions
Actions
Actions
Actions
Actions

Grants and funding

MC_PC_20059/MRC_/Medical Research Council/United Kingdom

LinkOut - more resources

Full Text Sources
Other Literature Sources
- The Lens - Patent Citations Database
Medical
- ClinicalTrials.gov

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI

Collaborators

Affiliations

Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI

Authors

Collaborators

Affiliations

Abstract

Conflict of interest statement

Figures

References

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources

Other Literature Sources

Medical