PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods

doi:10.1136/bmj-2024-082505

. 2025 Mar 24:388:e082505.

doi: 10.1136/bmj-2024-082505.

PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods

Karel G M Moons¹, Johanna A A Damen^{2

3}, Tabea Kaul², Lotty Hooft^{2

3}, Constanza Andaur Navarro², Paula Dhiman⁴, Andrew L Beam⁵, Ben Van Calster^{6

7}, Leo Anthony Celi^{8

9}, Spiros Denaxas^{10

11}, Alastair K Denniston¹², Marzyeh Ghassemi¹³, Georg Heinze¹⁴, André Pascal Kengne¹⁵, Lena Maier-Hein^{16

17}, Xiaoxuan Liu^{12

18

19

20}, Patricia Logullo⁴, Melissa D McCradden²¹, Nan Liu²², Lauren Oakden-Rayner²³, Karandeep Singh²⁴, Daniel S Ting^{22

25}, Laure Wynants^{6

26}, Bada Yang^{2

3}, Johannes B Reitsma², Richard D Riley^{19

20}, Gary S Collins⁴, Maarten van Smeden²

Affiliations

¹ Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, 3508 GA Utrecht, Netherlands k.g.m.moons@umcutrecht.nl.
² Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, 3508 GA Utrecht, Netherlands.
³ Cochrane Netherlands, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands.
⁴ Centre for Statistics in Medicine, UK EQUATOR Centre, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK.
⁵ Department of Epidemiology, Harvard T H Chan School of Public Health, Boston, MA, USA.
⁶ Department of Development and Regeneration, KU Leuven, Leuven, Belgium.
⁷ Leuven Unit for Health Technology Assessment Research (LUHTAR), KU Leuven, Leuven, Belgium.
⁸ Department of Biostatistics, Harvard T H Chan School of Public Health, Boston, MA, USA.
⁹ Laboratory for Computational Physiology, Massachusetts Institute of Technology, Cambridge, MA, USA.
¹⁰ Institute of Health Informatics, University College London, London, UK.
¹¹ British Heart Foundation Data Science Centre, Health Data Research Centre UK, London, United Kingdom.
¹² College of Medicine and Health, University of Birmingham, Birmingham, UK.
¹³ Department of Electrical Engineering and Computer Science, Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, MA, USA.
¹⁴ Institute of Clinical Biometrics, Centre for Medical Data Science, Medical University of Vienna, Vienna, Austria.
¹⁵ Department of Medicine, University of Cape Town, Cape Town, South Africa.
¹⁶ Division of Intelligent Medical Systems, German Cancer Research Centre (DKFZ), Heidelberg, Germany.
¹⁷ National Centre for Tumour Diseases (NCT) Heidelberg, Heidelberg, Germany.
¹⁸ University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK.
¹⁹ School of Health Sciences, College of Medicine and Health, University of Birmingham, Birmingham, UK.
²⁰ NIHR Birmingham Biomedical Research Centre, Birmingham, UK.
²¹ Department of Bioethics, The Hospital for Sick Children, Toronto, ON, Canada.
²² Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, Singapore.
²³ Australian Institute for Machine Learning, University of Adelaide, Adelaide, SA, Australia.
²⁴ Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor, MI, USA.
²⁵ AI Office, Singapore Health Service, Duke-NUS Medical School, Singapore, Singapore.
²⁶ Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, Maastricht, Netherlands.

PMID: 40127903
PMCID: PMC11931409
DOI: 10.1136/bmj-2024-082505

PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods

Karel G M Moons et al. BMJ. 2025.

. 2025 Mar 24:388:e082505.

doi: 10.1136/bmj-2024-082505.

Authors

Affiliations

¹ Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, 3508 GA Utrecht, Netherlands k.g.m.moons@umcutrecht.nl.
² Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, 3508 GA Utrecht, Netherlands.
³ Cochrane Netherlands, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands.
⁴ Centre for Statistics in Medicine, UK EQUATOR Centre, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK.
⁵ Department of Epidemiology, Harvard T H Chan School of Public Health, Boston, MA, USA.
⁶ Department of Development and Regeneration, KU Leuven, Leuven, Belgium.
⁷ Leuven Unit for Health Technology Assessment Research (LUHTAR), KU Leuven, Leuven, Belgium.
⁸ Department of Biostatistics, Harvard T H Chan School of Public Health, Boston, MA, USA.
⁹ Laboratory for Computational Physiology, Massachusetts Institute of Technology, Cambridge, MA, USA.
¹⁰ Institute of Health Informatics, University College London, London, UK.
¹¹ British Heart Foundation Data Science Centre, Health Data Research Centre UK, London, United Kingdom.
¹² College of Medicine and Health, University of Birmingham, Birmingham, UK.
¹³ Department of Electrical Engineering and Computer Science, Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, MA, USA.
¹⁴ Institute of Clinical Biometrics, Centre for Medical Data Science, Medical University of Vienna, Vienna, Austria.
¹⁵ Department of Medicine, University of Cape Town, Cape Town, South Africa.
¹⁶ Division of Intelligent Medical Systems, German Cancer Research Centre (DKFZ), Heidelberg, Germany.
¹⁷ National Centre for Tumour Diseases (NCT) Heidelberg, Heidelberg, Germany.
¹⁸ University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK.
¹⁹ School of Health Sciences, College of Medicine and Health, University of Birmingham, Birmingham, UK.
²⁰ NIHR Birmingham Biomedical Research Centre, Birmingham, UK.
²¹ Department of Bioethics, The Hospital for Sick Children, Toronto, ON, Canada.
²² Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, Singapore.
²³ Australian Institute for Machine Learning, University of Adelaide, Adelaide, SA, Australia.
²⁴ Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor, MI, USA.
²⁵ AI Office, Singapore Health Service, Duke-NUS Medical School, Singapore, Singapore.
²⁶ Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, Maastricht, Netherlands.

PMID: 40127903
PMCID: PMC11931409
DOI: 10.1136/bmj-2024-082505

Abstract

The Prediction model Risk Of Bias ASsessment Tool (PROBAST) is used to assess the quality, risk of bias, and applicability of prediction models or algorithms and of prediction model/algorithm studies. Since PROBAST’s introduction in 2019, much progress has been made in the methodology for prediction modelling and in the use of artificial intelligence, including machine learning, techniques. An update to PROBAST-2019 is thus needed. This article describes the development of PROBAST+AI. PROBAST+AI consists of two distinctive parts: model development and model evaluation. For model development, PROBAST+AI users assess quality and applicability using 16 targeted signalling questions. For model evaluation, PROBAST+AI users assess the risk of bias and applicability using 18 targeted signalling questions. Both parts contain four domains: participants and data sources, predictors, outcome, and analysis. Applicability of the prediction model is rated for the participants and data sources, predictors, and outcome domains. PROBAST+AI may replace the original PROBAST tool and allows all key stakeholders (eg, model developers, AI companies, researchers, editors, reviewers, healthcare professionals, guideline developers, and policy organisations) to examine the quality, risk of bias, and applicability of any type of prediction model in the healthcare sector, irrespective of whether regression modelling or AI techniques are used.

PubMed Disclaimer

Conflict of interest statement

Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/disclosure-of-interest/ and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years, no other relationships or activities that could appear to have influenced the submitted work. KGMM is director of Health Innovation Netherlands (HI-NL), editor in chief of BMC Diagnostic and Prognostic Research, and principal investigator and author of the “Guidance for high quality AI in healthcare” (https://guideline-ai-healthcare.com). GSC is director of the UK EQUATOR Centre, editor in chief of BMC Diagnostic and Prognostic Research, and a statistical editor for The BMJ. PL is a meta-researcher with the UK EQUATOR Centre. RR is a statistical editor for The BMJ and receives royalties for two textbooks: Prognosis Research in Healthcare and Individual Participant Data Meta-Analysis.

Cited by

Implementing Artificial Intelligence in Critical Care Medicine: a consensus of 22.
Cecconi M, Greco M, Shickel B, Angus DC, Bailey H, Bignami E, Calandra T, Celi LA, Einav S, Elbers P, Ercole A, Gómez H, Gong MN, Komorowski M, Liu V, Park S, Sarwal A, Seymour CW, Zampieri FG, Taccone FS, Vincent JL, Bihorac A. Cecconi M, et al. Crit Care. 2025 Jul 8;29(1):290. doi: 10.1186/s13054-025-05532-2. Crit Care. 2025. PMID: 40629428 Free PMC article. Review.
Letter to the editor concerning "A multimodal machine learning model integrating clinical and MRI data for predicting neurological outcomes following surgical treatment for cervical spinal cord injury" by T.Shimizu, et al. (Eur Spine J [2025]: doi: 10.1007/s00586-025-08873-2).
Yu W, Chen Y. Yu W, et al. Eur Spine J. 2025 Jun 9. doi: 10.1007/s00586-025-09017-2. Online ahead of print. Eur Spine J. 2025. PMID: 40488871 No abstract available.
Development of a consensus extension of the estimands framework for cluster randomised trials (CRT-estimands): results from an international Delphi study.
Kahan BC, Bahti M, Bi D, Bretz F, Collins GS, Copas A, Harhay MO, Li F, Auriemma CL. Kahan BC, et al. medRxiv [Preprint]. 2025 Jul 3:2025.07.03.25330799. doi: 10.1101/2025.07.03.25330799. medRxiv. 2025. PMID: 40630601 Free PMC article. Preprint.
Machine learning-based prediction of short- and long-term mortality for shared decision-making in older hip fracture patients: the Dutch Hip Fracture Audit algorithms in 74,396 cases.
Dijkstra H, Parsons CS, Bremen HV, Willems HC, De Hond AAH, Van Munster BC, Doornberg JN, Oosterhoff JHF. Dijkstra H, et al. Acta Orthop. 2025 Jul 7;96:521-528. doi: 10.2340/17453674.2025.44248. Acta Orthop. 2025. PMID: 40622227 Free PMC article.
Artificial intelligence tool development: what clinicians need to know?
Chew BH, Ngiam KY. Chew BH, et al. BMC Med. 2025 Apr 24;23(1):244. doi: 10.1186/s12916-025-04076-0. BMC Med. 2025. PMID: 40275334 Free PMC article. Review.

See all "Cited by" articles

References

1. Hippisley-Cox J, Coupland CAC, Bafadhel M, et al. . Development and validation of a new algorithm for improved cardiovascular risk prediction. Nat Med 2024;30:1440-7. 10.1038/s41591-024-02905-y - DOI - PMC - PubMed
1. Damen JAAG, Hooft L, Schuit E, et al. . Prediction models for cardiovascular disease risk in the general population: systematic review. BMJ 2016;353:i2416. 10.1136/bmj.i2416 - DOI - PMC - PubMed
1. Bellou V, Belbasis L, Konstantinidis AK, Tzoulaki I, Evangelou E. Prognostic models for outcome prediction in patients with chronic obstructive pulmonary disease: systematic review and critical appraisal. BMJ 2019;367:l5358. 10.1136/bmj.l5358 - DOI - PMC - PubMed
1. de Munter L, Polinder S, Lansink KW, Cnossen MC, Steyerberg EW, de Jongh MA. Mortality prediction models in the general trauma population: A systematic review. Injury 2017;48:221-9. 10.1016/j.injury.2016.12.009 - DOI - PubMed
1. Wynants L, Van Calster B, Collins GS, et al. . Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ 2020;369:m1328. 10.1136/bmj.m1328 - DOI - PMC - PubMed

LinkOut - more resources

Full Text Sources
- HighWire
- PubMed Central

[1] Hippisley-Cox J, Coupland CAC, Bafadhel M, et al. . Development and validation of a new algorithm for improved cardiovascular risk prediction. Nat Med 2024;30:1440-7. 10.1038/s41591-024-02905-y - DOI - PMC - PubMed

[2] Hippisley-Cox J, Coupland CAC, Bafadhel M, et al. . Development and validation of a new algorithm for improved cardiovascular risk prediction. Nat Med 2024;30:1440-7. 10.1038/s41591-024-02905-y - DOI - PMC - PubMed

[3] Damen JAAG, Hooft L, Schuit E, et al. . Prediction models for cardiovascular disease risk in the general population: systematic review. BMJ 2016;353:i2416. 10.1136/bmj.i2416 - DOI - PMC - PubMed

[4] Damen JAAG, Hooft L, Schuit E, et al. . Prediction models for cardiovascular disease risk in the general population: systematic review. BMJ 2016;353:i2416. 10.1136/bmj.i2416 - DOI - PMC - PubMed

[5] Bellou V, Belbasis L, Konstantinidis AK, Tzoulaki I, Evangelou E. Prognostic models for outcome prediction in patients with chronic obstructive pulmonary disease: systematic review and critical appraisal. BMJ 2019;367:l5358. 10.1136/bmj.l5358 - DOI - PMC - PubMed

[6] Bellou V, Belbasis L, Konstantinidis AK, Tzoulaki I, Evangelou E. Prognostic models for outcome prediction in patients with chronic obstructive pulmonary disease: systematic review and critical appraisal. BMJ 2019;367:l5358. 10.1136/bmj.l5358 - DOI - PMC - PubMed

[7] de Munter L, Polinder S, Lansink KW, Cnossen MC, Steyerberg EW, de Jongh MA. Mortality prediction models in the general trauma population: A systematic review. Injury 2017;48:221-9. 10.1016/j.injury.2016.12.009 - DOI - PubMed

[8] de Munter L, Polinder S, Lansink KW, Cnossen MC, Steyerberg EW, de Jongh MA. Mortality prediction models in the general trauma population: A systematic review. Injury 2017;48:221-9. 10.1016/j.injury.2016.12.009 - DOI - PubMed

[9] Wynants L, Van Calster B, Collins GS, et al. . Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ 2020;369:m1328. 10.1136/bmj.m1328 - DOI - PMC - PubMed

[10] Wynants L, Van Calster B, Collins GS, et al. . Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ 2020;369:m1328. 10.1136/bmj.m1328 - DOI - PMC - PubMed

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods

Affiliations

PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods

Authors

Affiliations

Abstract

Conflict of interest statement

Similar articles

Cited by

References

LinkOut - more resources

Full Text Sources

Abstract

Conflict of interest statement

Similar articles

Cited by

References

Related information

LinkOut - more resources

Full Text Sources