Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI

Baptiste Vasey^{1

2

3}, Myura Nagendran⁴, Bruce Campbell^{5

6}, David A Clifton⁷, Gary S Collins⁸, Spiros Denaxas^{9

10

11

12}, Alastair K Denniston^{13

14

15}, Livia Faes¹⁵, Bart Geerts¹⁶, Mudathir Ibrahim^{17

18}, Xiaoxuan Liu^{13

14}, Bilal A Mateen^{9

19

20}, Piyush Mathur²¹, Melissa D McCradden^{22

23}, Lauren Morgan²⁴, Johan Ordish²⁵, Campbell Rogers²⁶, Suchi Saria^{27

28}, Daniel S W Ting^{29

30}, Peter Watkinson^{31

32}, Wim Weber³³, Peter Wheatstone³⁴, Peter McCulloch¹⁷; DECIDE-AI expert group

Collaborators, Affiliations

Collaborators

DECIDE-AI expert group:
Aaron Y Lee, Alan G Fraser, Ali Connell, Alykhan Vira, Andre Esteva, Andrew D Althouse, Andrew L Beam, Anne de Hond, Anne-Laure Boulesteix, Anthony Bradlow, Ari Ercole, Arsenio Paez, Athanasios Tsanas, Barry Kirby, Ben Glocker, Carmelo Velardo, Chang Min Park, Charisma Hehakaya, Chris Baber, Chris Paton, Christian Johner, Christopher J Kelly, Christopher J Vincent, Christopher Yau, Clare McGenity, Constantine Gatsonis, Corinne Faivre-Finn, Crispin Simon, Danielle Sent, Danilo Bzdok, Darren Treanor, David C Wong, David F Steiner, David Higgins, Dawn Benson, Declan P O'Regan, Dinesh V Gunasekaran, Dominic Danks, Emanuele Neri, Evangelia Kyrimi, Falk Schwendicke, Farah Magrabi, Frances Ives, Frank E Rademakers, George E Fowler, Giuseppe Frau, H D Jeffry Hogg, Hani J Marcus, Heang-Ping Chan, Henry Xiang, Hugh F McIntyre, Hugh Harvey, Hyungjin Kim, Ibrahim Habli, James C Fackler, James Shaw, Janet Higham, Jared M Wohlgemut, Jaron Chong, Jean-Emmanuel Bibault, Jérémie F Cohen, Jesper Kers, Jessica Morley, Joachim Krois, Joao Monteiro, Joel Horovitz, John Fletcher, Jonathan Taylor, Jung Hyun Yoon, Karandeep Singh, Karel G M Moons, Kassandra Karpathakis, Ken Catchpole, Kerenza Hood, Konstantinos Balaskas, Konstantinos Kamnitsas, Laura Militello, Laure Wynants, Lauren Oakden-Rayner, Laurence B Lovat, Luc J M Smits, Ludwig C Hinske, M Khair ElZarrad, Maarten van Smeden, Mara Giavina-Bianchi, Mark Daley, Mark P Sendak, Mark Sujan, Maroeska Rovers, Matthew DeCamp, Matthew Woodward, Matthieu Komorowski, Max Marsden, Maxine Mackintosh, Michael D Abramoff, Miguel Ángel Armengol de la Hoz, Neale Hambidge, Neil Daly, Niels Peek, Oliver Redfern, Omer F Ahmad, Patrick M Bossuyt, Pearse A Keane, Pedro N P Ferreira, Petra Schnell-Inderst, Pietro Mascagni, Prokar Dasgupta, Pujun Guan, Rachel Barnett, Rawen Kader, Reena Chopra, Ritse M Mann, Rupa Sarkar, Saana M Mäenpää, Samuel G Finlayson, Sarah Vollam, Sebastian J Vollmer, Seong Ho Park, Shakir Laher, Shalmali Joshi, Siri L van der Meijden, Susan C Shelmerdine, Tien-En Tan, Tom J W Stocker, Valentina Giannini, Vince I Madai, Virginia Newcombe, Wei Yan Ng, Wendy A Rogers, William Ogallo, Yoonyoung Park, Zane B Perkins

Affiliations

¹ Nuffield Department of Surgical Sciences, University of Oxford, Oxford, UK. baptiste.vasey@gmail.com.
² Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK. baptiste.vasey@gmail.com.
³ Critical Care Research Group, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK. baptiste.vasey@gmail.com.
⁴ UKRI Centre for Doctoral Training in AI for Healthcare, Imperial College London, London, UK.
⁵ University of Exeter Medical School, Exeter, UK.
⁶ Royal Devon and Exeter Hospital, Exeter, UK.
⁷ Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK.
⁸ Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology & Musculoskeletal Sciences, University of Oxford, Oxford, UK.
⁹ Institute of Health Informatics, University College London, London, UK.
¹⁰ British Heart Foundation Data Science Centre, London, UK.
¹¹ Health Data Research UK, London, UK.
¹² UCL Hospitals Biomedical Research Centre, London, UK.
¹³ University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK.
¹⁴ Academic Unit of Ophthalmology, Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK.
¹⁵ Moorfields Eye Hospital NHS Foundation Trust, London, UK.
¹⁶ Healthplus.ai-R&D BV, Amsterdam, The Netherlands.
¹⁷ Nuffield Department of Surgical Sciences, University of Oxford, Oxford, UK.
¹⁸ Department of Surgery, Maimonides Medical Center, Brooklyn, NY, USA.
¹⁹ The Wellcome Trust, London, UK.
²⁰ The Alan Turing Institute, London, UK.
²¹ Department of General Anesthesiology, Anesthesiology Institute, Cleveland Clinic, Cleveland, OH, USA.
²² The Hospital for Sick Children, Toronto ON, Canada.
²³ Dalla Lana School of Public Health, University of Toronto, Toronto ON, Canada.
²⁴ Morgan Human Systems Ltd, Shrewsbury, UK.
²⁵ Medicines and Healthcare products Regulatory Agency, London, UK.
²⁶ HeartFlow Inc., Redwood City, CA, USA.
²⁷ Departments of Computer Science, Statistics, and Health Policy, and Division of Informatics, Johns Hopkins University, Baltimore, MD, USA.
²⁸ Bayesian Health, New York, NY, USA.
²⁹ Singapore National Eye Center, Singapore Eye Research Institute, Singapore, Singapore.
³⁰ Duke-NUS Medical School, National University of Singapore, Singapore, Singapore.
³¹ Critical Care Research Group, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK.
³² NIHR Biomedical Research Centre Oxford, Oxford University Hospitals NHS Trust, Oxford, UK.
³³ The BMJ, London, UK.
³⁴ School of Medicine, University of Leeds, Leeds, UK.

PMID: 35585198
DOI: 10.1038/s41591-022-01772-9

Review

Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI

Baptiste Vasey et al. Nat Med. 2022 May.

. 2022 May;28(5):924-933.

doi: 10.1038/s41591-022-01772-9. Epub 2022 May 18.

Authors

Collaborators

DECIDE-AI expert group:
Aaron Y Lee, Alan G Fraser, Ali Connell, Alykhan Vira, Andre Esteva, Andrew D Althouse, Andrew L Beam, Anne de Hond, Anne-Laure Boulesteix, Anthony Bradlow, Ari Ercole, Arsenio Paez, Athanasios Tsanas, Barry Kirby, Ben Glocker, Carmelo Velardo, Chang Min Park, Charisma Hehakaya, Chris Baber, Chris Paton, Christian Johner, Christopher J Kelly, Christopher J Vincent, Christopher Yau, Clare McGenity, Constantine Gatsonis, Corinne Faivre-Finn, Crispin Simon, Danielle Sent, Danilo Bzdok, Darren Treanor, David C Wong, David F Steiner, David Higgins, Dawn Benson, Declan P O'Regan, Dinesh V Gunasekaran, Dominic Danks, Emanuele Neri, Evangelia Kyrimi, Falk Schwendicke, Farah Magrabi, Frances Ives, Frank E Rademakers, George E Fowler, Giuseppe Frau, H D Jeffry Hogg, Hani J Marcus, Heang-Ping Chan, Henry Xiang, Hugh F McIntyre, Hugh Harvey, Hyungjin Kim, Ibrahim Habli, James C Fackler, James Shaw, Janet Higham, Jared M Wohlgemut, Jaron Chong, Jean-Emmanuel Bibault, Jérémie F Cohen, Jesper Kers, Jessica Morley, Joachim Krois, Joao Monteiro, Joel Horovitz, John Fletcher, Jonathan Taylor, Jung Hyun Yoon, Karandeep Singh, Karel G M Moons, Kassandra Karpathakis, Ken Catchpole, Kerenza Hood, Konstantinos Balaskas, Konstantinos Kamnitsas, Laura Militello, Laure Wynants, Lauren Oakden-Rayner, Laurence B Lovat, Luc J M Smits, Ludwig C Hinske, M Khair ElZarrad, Maarten van Smeden, Mara Giavina-Bianchi, Mark Daley, Mark P Sendak, Mark Sujan, Maroeska Rovers, Matthew DeCamp, Matthew Woodward, Matthieu Komorowski, Max Marsden, Maxine Mackintosh, Michael D Abramoff, Miguel Ángel Armengol de la Hoz, Neale Hambidge, Neil Daly, Niels Peek, Oliver Redfern, Omer F Ahmad, Patrick M Bossuyt, Pearse A Keane, Pedro N P Ferreira, Petra Schnell-Inderst, Pietro Mascagni, Prokar Dasgupta, Pujun Guan, Rachel Barnett, Rawen Kader, Reena Chopra, Ritse M Mann, Rupa Sarkar, Saana M Mäenpää, Samuel G Finlayson, Sarah Vollam, Sebastian J Vollmer, Seong Ho Park, Shakir Laher, Shalmali Joshi, Siri L van der Meijden, Susan C Shelmerdine, Tien-En Tan, Tom J W Stocker, Valentina Giannini, Vince I Madai, Virginia Newcombe, Wei Yan Ng, Wendy A Rogers, William Ogallo, Yoonyoung Park, Zane B Perkins

Affiliations

¹ Nuffield Department of Surgical Sciences, University of Oxford, Oxford, UK. baptiste.vasey@gmail.com.
² Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK. baptiste.vasey@gmail.com.
³ Critical Care Research Group, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK. baptiste.vasey@gmail.com.
⁴ UKRI Centre for Doctoral Training in AI for Healthcare, Imperial College London, London, UK.
⁵ University of Exeter Medical School, Exeter, UK.
⁶ Royal Devon and Exeter Hospital, Exeter, UK.
⁷ Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK.
⁸ Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology & Musculoskeletal Sciences, University of Oxford, Oxford, UK.
⁹ Institute of Health Informatics, University College London, London, UK.
¹⁰ British Heart Foundation Data Science Centre, London, UK.
¹¹ Health Data Research UK, London, UK.
¹² UCL Hospitals Biomedical Research Centre, London, UK.
¹³ University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK.
¹⁴ Academic Unit of Ophthalmology, Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK.
¹⁵ Moorfields Eye Hospital NHS Foundation Trust, London, UK.
¹⁶ Healthplus.ai-R&D BV, Amsterdam, The Netherlands.
¹⁷ Nuffield Department of Surgical Sciences, University of Oxford, Oxford, UK.
¹⁸ Department of Surgery, Maimonides Medical Center, Brooklyn, NY, USA.
¹⁹ The Wellcome Trust, London, UK.
²⁰ The Alan Turing Institute, London, UK.
²¹ Department of General Anesthesiology, Anesthesiology Institute, Cleveland Clinic, Cleveland, OH, USA.
²² The Hospital for Sick Children, Toronto ON, Canada.
²³ Dalla Lana School of Public Health, University of Toronto, Toronto ON, Canada.
²⁴ Morgan Human Systems Ltd, Shrewsbury, UK.
²⁵ Medicines and Healthcare products Regulatory Agency, London, UK.
²⁶ HeartFlow Inc., Redwood City, CA, USA.
²⁷ Departments of Computer Science, Statistics, and Health Policy, and Division of Informatics, Johns Hopkins University, Baltimore, MD, USA.
²⁸ Bayesian Health, New York, NY, USA.
²⁹ Singapore National Eye Center, Singapore Eye Research Institute, Singapore, Singapore.
³⁰ Duke-NUS Medical School, National University of Singapore, Singapore, Singapore.
³¹ Critical Care Research Group, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK.
³² NIHR Biomedical Research Centre Oxford, Oxford University Hospitals NHS Trust, Oxford, UK.
³³ The BMJ, London, UK.
³⁴ School of Medicine, University of Leeds, Leeds, UK.

PMID: 35585198
DOI: 10.1038/s41591-022-01772-9

Erratum in

Publisher Correction: Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI.
Vasey B, Nagendran M, Campbell B, Clifton DA, Collins GS, Denaxas S, Denniston AK, Faes L, Geerts B, Ibrahim M, Liu X, Mateen BA, Mathur P, McCradden MD, Morgan L, Ordish J, Rogers C, Saria S, Ting DSW, Watkinson P, Weber W, Wheatstone P, McCulloch P; DECIDE-AI expert group. Vasey B, et al. Nat Med. 2022 Oct;28(10):2218. doi: 10.1038/s41591-022-01951-8. Nat Med. 2022. PMID: 35962208 No abstract available.

Abstract

A growing number of artificial intelligence (AI)-based clinical decision support systems are showing promising performance in preclinical, in silico evaluation, but few have yet demonstrated real benefit to patient care. Early-stage clinical evaluation is important to assess an AI system's actual clinical performance at small scale, ensure its safety, evaluate the human factors surrounding its use and pave the way to further large-scale trials. However, the reporting of these early studies remains inadequate. The present statement provides a multi-stakeholder, consensus-based reporting guideline for the Developmental and Exploratory Clinical Investigations of DEcision support systems driven by Artificial Intelligence (DECIDE-AI). We conducted a two-round, modified Delphi process to collect and analyze expert opinion on the reporting of early clinical evaluation of AI systems. Experts were recruited from 20 pre-defined stakeholder categories. The final composition and wording of the guideline was determined at a virtual consensus meeting. The checklist and the Explanation & Elaboration (E&E) sections were refined based on feedback from a qualitative evaluation process. In total, 123 experts participated in the first round of Delphi, 138 in the second round, 16 in the consensus meeting and 16 in the qualitative evaluation. The DECIDE-AI reporting guideline comprises 17 AI-specific reporting items (made of 28 subitems) and ten generic reporting items, with an E&E paragraph provided for each. Through consultation and consensus with a range of stakeholders, we developed a guideline comprising key items that should be reported in early-stage clinical studies of AI-based decision support systems in healthcare. By providing an actionable checklist of minimal reporting items, the DECIDE-AI guideline will facilitate the appraisal of these studies and replicability of their findings.

PubMed Disclaimer

References

1. Skivington, K. et al. A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. Br. Med. J. 374, n2061 (2021). - DOI
1. Liu, X. et al. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit. Health 1, e271–e297 (2019). - PubMed - DOI
1. Vasey, B. et al. Association of clinician diagnostic performance with machine learning-based decision support systems: a systematic review. JAMA Netw. Open 4, e211276 (2021).
1. Freeman, K. et al. Use of artificial intelligence for image analysis in breast cancer screening programmes: systematic review of test accuracy. Br. Med. J. 374, n1872 (2021). - DOI
1. Keane, P. A. & Topol, E. J. With an eye to AI and autonomous diagnosis. NPJ Digital Med. 1, 40 (2018). - DOI

Publication types

Actions
Actions
Actions
Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions

Grants and funding

LinkOut - more resources

Full Text Sources
- Nature Publishing Group
Other Literature Sources
- The Lens - Patent Citations Database

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI

Collaborators

Affiliations

Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI

Authors

Collaborators

Affiliations

Erratum in

Abstract

References

Publication types

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources

Other Literature Sources