Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Review
. 2025 May 23;3(3):100228.
doi: 10.1016/j.mcpdig.2025.100228. eCollection 2025 Sep.

Implementation and Updating of Clinical Prediction Models: A Systematic Review

Alexander Saelmans  1 , Tom Seinen  1 , Victor Pera  1 , Aniek F Markus  1 , Egill Fridgeirsson  1 , Luis H John  1 , Lieke Schiphof-Godart  1 , Peter Rijnbeek  1 , Jenna Reps  1   2 , Ross Williams  1
Affiliations
Review

Implementation and Updating of Clinical Prediction Models: A Systematic Review

Alexander Saelmans et al. Mayo Clin Proc Digit Health. .

Abstract

Objective: To summarize the implementation approaches and updating methods of clinically implemented models and consecutively advise researchers on the implementation and updating.

Patients and methods: We included studies describing the implementation of prognostic binary prediction models in a clinical setting. We retrieved articles from Embase, Medline, and Web of Science from January 1, 2010, to January 1, 2024. We performed data extraction, based on Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis and Prediction Model Risk of Bias Assessment guidelines, and summarized.

Results: The search yielded 1872 articles. Following screening, 37 articles, describing 56 prediction models, were eligible for inclusion. The overall risk of bias was high in 86% of publications. In model development and internal validation, 32% of the models was assessed for calibration. External validation was performed for 27% of the models. Most models were implemented into the hospital information system (63%), followed by a web application (32%) and a patient decision aid tool (5%). Moreover, 13% of models have been updated following implementation.

Conclusion: Impact assessments generally showed successful model implementation and the ability to improve patient care, despite not fully adhering to prediction modeling best practice. Both impact assessment and updating could play a key role in identifying and lowering bias in models.

PubMed Disclaimer

Conflict of interest statement

Dr Reps is an employee and shareholder of Johnson & Johnson. All authors work for a research group at Erasmus University Medical Center that receives/received an unconditional grant for methodological research by Johnson & Johnson. The grant is for the institute.

Figures

Figure 1
Figure 1
Pipeline of clinical prediction models. Model development and internal validation, assessment of performance in the same setting as the derivation data; external validation, assessment of performance in different database; impact assessment, assessment of performance in a clinical setting; implementation, eg, hospital information system, web-based application, and patient decision aid tool; update, simple (recalibration or revision), extension, meta-model, and dynamic.
Figure 2
Figure 2
(A) Trend in modeling methods, publication year of implementation article. (B) Boxplots of number of features per modeling method. (C) PROBAST risk of bias assessment of the included prediction model studies. Other—2 absolute risk models, 1 Bayesian neural network, 1 deep convolutional neural network, and 1 decision forest model. One random forest number of feature outlier of 9614 features is not depicted in (B).
Figure 3
Figure 3
Sankey diagram of implementation, impact assessment, and clinical improvement of models.
See this image and copyright information in PMC

References

    1. Stiggelbout A.M., van der Weijden T., de Wit M.P., et al. Shared decision making: really putting patients at the centre of healthcare. BMJ. 2012;344 doi: 10.1136/bmj.e256. - DOI - PubMed
    1. Harrell F.J. Springer; 2001. Regression Modeling Strategies With Applications to Linear Models, Logistic Regression, and Survival Analysis.
    1. Moons K.G.M., Royston P., Vergouwe Y., Grobbee D.E., Altman D.G. Prognosis and prognostic research: what, why and how? BMJ. 2009;338 doi: 10.1136/bmj.b375. - DOI - PubMed
    1. Steyerberg E. Springer; 2009. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating.
    1. Steyerberg E.W., Moons K.G., van der Windt D.A., et al. Prognosis Research Strategy (PROGRESS) 3: prognostic model research. PLoS Med. 2013;10(2) doi: 10.1371/journal.pmed.1001381. - DOI - PMC - PubMed

LinkOut - more resources