. 2021 Jan:129:138-150.

doi: 10.1016/j.jclinepi.2020.09.018. Epub 2020 Sep 24.

GRADE Guidelines 30: the GRADE approach to assessing the certainty of modeled evidence-An overview in the context of health decision-making

Jan L Brozek¹, Carlos Canelo-Aybar², Elie A Akl³, James M Bowen⁴, John Bucher⁵, Weihsueh A Chiu⁶, Mark Cronin⁷, Benjamin Djulbegovic⁸, Maicon Falavigna⁹, Gordon H Guyatt¹, Ami A Gordon¹⁰, Michele Hilton Boon¹¹, Raymond C W Hutubessy¹², Manuela A Joore¹³, Vittal Katikireddi¹¹, Judy LaKind¹⁴, Miranda Langendam¹⁵, Veena Manja¹⁶, Kristen Magnuson¹⁰, Alexander G Mathioudakis¹⁷, Joerg Meerpohl¹⁸, Dominik Mertz¹⁹, Roman Mezencev²⁰, Rebecca Morgan¹⁹, Gian Paolo Morgano²¹, Reem Mustafa²², Martin O'Flaherty²³, Grace Patlewicz²⁴, John J Riva²⁵, Margarita Posso²⁶, Andrew Rooney⁵, Paul M Schlosser²⁰, Lisa Schwartz¹⁹, Ian Shemilt²⁷, Jean-Eric Tarride²⁸, Kristina A Thayer²⁹, Katya Tsaioun³⁰, Luke Vale³¹, John Wambaugh²⁴, Jessica Wignall¹⁰, Ashley Williams¹⁰, Feng Xie¹⁹, Yuan Zhang³², Holger J Schünemann¹; GRADE Working Group

Affiliations

¹ Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Department of Medicine, McMaster University, Hamilton, Ontario, Canada; McMaster GRADE Centre & Michael DeGroote Cochrane Canada Centre, McMaster University, Hamilton, Ontario, Canada.
² Department of Paediatrics, Obstetrics and Gynaecology, Preventive Medicine, and Public Health. PhD Programme in Methodology of Biomedical Research and Public Health. Universitat Autònoma de Barcelona, Bellaterra, Spain; Iberoamerican Cochrane Center, Biomedical Research Institute (IIB Sant Pau-CIBERESP), Barcelona, Spain.
³ Department of Internal Medicine, American University of Beirut, Beirut, Lebanon.
⁴ Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Toronto Health Economics and Technology Assessment (THETA) Collaborative, Toronto, Ontario, Canada.
⁵ National Toxicology Program, National Institute of Environmental Health Sciences, Durham, NC, USA.
⁶ Department of Veterinary Integrative Biosciences, Texas A&M University, College Station, TX, USA.
⁷ School of Pharmacy and Chemistry, Liverpool John Moores University, Liverpool, UK.
⁸ Center for Evidence-Based Medicine and Health Outcome Research, Morsani College of Medicine, University of South Florida, Tampa, Florida, USA.
⁹ Institute for Education and Research, Hospital Moinhos de Vento, Porto Alegre, Rio Grande do Sul, Brazil.
¹⁰ ICF International, Durham, NC, USA.
¹¹ Institute of Health & Wellbeing, University of Glasgow, Glasgow, UK.
¹² Department of Immunization, Vaccines and Biologicals, World Health Organization, Geneva, Switzerland.
¹³ Clinical Epidemiology and Medical Technology Assessment, Maastricht University Medical Centre+, Maastricht, the Netherlands.
¹⁴ LaKind Associates, LLC, Catonsville, MD, USA; Department of Epidemiology and Public Health, University of Maryland School of Medicine, Baltimore, MD, USA.
¹⁵ Department of Clinical Epidemiology, Biostatistics, and Bioinformatics, Academic Medical Center, University of Amsterdam, Amsterdam, the Netherlands.
¹⁶ Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Department of Surgery, University of California Davis, Sacramento, CA, USA; Department of Medicine, Department of Veterans Affairs, Northern California Health Care System, Mather, CA, USA.
¹⁷ Division of Infection, Immunity and Respiratory Medicine, University Hospital of South Manchester, University of Manchester, Manchester, UK.
¹⁸ Institute for Evidence in Medicine, Medical Center, University of Freiburg, Freiburg-am-Breisgau, Germany; Cochrane Germany, Freiburg-am-Breisgau, Germany.
¹⁹ Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada.
²⁰ National Center for Environmental Assessment, U.S. Environmental Protection Agency, Washington, DC, USA.
²¹ Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; McMaster GRADE Centre & Michael DeGroote Cochrane Canada Centre, McMaster University, Hamilton, Ontario, Canada.
²² Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Department of Medicine, University of Kansas Medical Center, Kansas City, KS, USA.
²³ Institute of Population Health Sciences, University of Liverpool, Liverpool, UK.
²⁴ National Center for Computational Toxicology, U.S. Environmental Protection Agency, Durham, NC, USA.
²⁵ McMaster GRADE Centre & Michael DeGroote Cochrane Canada Centre, McMaster University, Hamilton, Ontario, Canada; Department of Family Medicine, McMaster University, Hamilton, Ontario, Canada.
²⁶ Iberoamerican Cochrane Center, Biomedical Research Institute (IIB Sant Pau-CIBERESP), Barcelona, Spain.
²⁷ EPPI-Centre, Institute of Education, University College London, London, UK.
²⁸ Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Programs for Assessment of Technology in Health, McMaster University, Hamilton, Ontario, Canada.
²⁹ Department of Medicine, Department of Veterans Affairs, Northern California Health Care System, Mather, CA, USA.
³⁰ Evidence-Based Toxicology Collaboration, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA.
³¹ Health Economics Group, Institute of Health and Society, Newcastle University, Newcastle upon Tyne, UK.
³² Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Health Quality Ontario, Toronto, Ontario, Canada.

PMID: 32980429
PMCID: PMC8514123
DOI: 10.1016/j.jclinepi.2020.09.018

GRADE Guidelines 30: the GRADE approach to assessing the certainty of modeled evidence-An overview in the context of health decision-making

Jan L Brozek et al. J Clin Epidemiol. 2021 Jan.

. 2021 Jan:129:138-150.

doi: 10.1016/j.jclinepi.2020.09.018. Epub 2020 Sep 24.

Authors

Affiliations

¹ Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Department of Medicine, McMaster University, Hamilton, Ontario, Canada; McMaster GRADE Centre & Michael DeGroote Cochrane Canada Centre, McMaster University, Hamilton, Ontario, Canada.
² Department of Paediatrics, Obstetrics and Gynaecology, Preventive Medicine, and Public Health. PhD Programme in Methodology of Biomedical Research and Public Health. Universitat Autònoma de Barcelona, Bellaterra, Spain; Iberoamerican Cochrane Center, Biomedical Research Institute (IIB Sant Pau-CIBERESP), Barcelona, Spain.
³ Department of Internal Medicine, American University of Beirut, Beirut, Lebanon.
⁴ Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Toronto Health Economics and Technology Assessment (THETA) Collaborative, Toronto, Ontario, Canada.
⁵ National Toxicology Program, National Institute of Environmental Health Sciences, Durham, NC, USA.
⁶ Department of Veterinary Integrative Biosciences, Texas A&M University, College Station, TX, USA.
⁷ School of Pharmacy and Chemistry, Liverpool John Moores University, Liverpool, UK.
⁸ Center for Evidence-Based Medicine and Health Outcome Research, Morsani College of Medicine, University of South Florida, Tampa, Florida, USA.
⁹ Institute for Education and Research, Hospital Moinhos de Vento, Porto Alegre, Rio Grande do Sul, Brazil.
¹⁰ ICF International, Durham, NC, USA.
¹¹ Institute of Health & Wellbeing, University of Glasgow, Glasgow, UK.
¹² Department of Immunization, Vaccines and Biologicals, World Health Organization, Geneva, Switzerland.
¹³ Clinical Epidemiology and Medical Technology Assessment, Maastricht University Medical Centre+, Maastricht, the Netherlands.
¹⁴ LaKind Associates, LLC, Catonsville, MD, USA; Department of Epidemiology and Public Health, University of Maryland School of Medicine, Baltimore, MD, USA.
¹⁵ Department of Clinical Epidemiology, Biostatistics, and Bioinformatics, Academic Medical Center, University of Amsterdam, Amsterdam, the Netherlands.
¹⁶ Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Department of Surgery, University of California Davis, Sacramento, CA, USA; Department of Medicine, Department of Veterans Affairs, Northern California Health Care System, Mather, CA, USA.
¹⁷ Division of Infection, Immunity and Respiratory Medicine, University Hospital of South Manchester, University of Manchester, Manchester, UK.
¹⁸ Institute for Evidence in Medicine, Medical Center, University of Freiburg, Freiburg-am-Breisgau, Germany; Cochrane Germany, Freiburg-am-Breisgau, Germany.
¹⁹ Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada.
²⁰ National Center for Environmental Assessment, U.S. Environmental Protection Agency, Washington, DC, USA.
²¹ Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; McMaster GRADE Centre & Michael DeGroote Cochrane Canada Centre, McMaster University, Hamilton, Ontario, Canada.
²² Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Department of Medicine, University of Kansas Medical Center, Kansas City, KS, USA.
²³ Institute of Population Health Sciences, University of Liverpool, Liverpool, UK.
²⁴ National Center for Computational Toxicology, U.S. Environmental Protection Agency, Durham, NC, USA.
²⁵ McMaster GRADE Centre & Michael DeGroote Cochrane Canada Centre, McMaster University, Hamilton, Ontario, Canada; Department of Family Medicine, McMaster University, Hamilton, Ontario, Canada.
²⁶ Iberoamerican Cochrane Center, Biomedical Research Institute (IIB Sant Pau-CIBERESP), Barcelona, Spain.
²⁷ EPPI-Centre, Institute of Education, University College London, London, UK.
²⁸ Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Programs for Assessment of Technology in Health, McMaster University, Hamilton, Ontario, Canada.
²⁹ Department of Medicine, Department of Veterans Affairs, Northern California Health Care System, Mather, CA, USA.
³⁰ Evidence-Based Toxicology Collaboration, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA.
³¹ Health Economics Group, Institute of Health and Society, Newcastle University, Newcastle upon Tyne, UK.
³² Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Health Quality Ontario, Toronto, Ontario, Canada.

PMID: 32980429
PMCID: PMC8514123
DOI: 10.1016/j.jclinepi.2020.09.018

Abstract

Objectives: The objective of the study is to present the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) conceptual approach to the assessment of certainty of evidence from modeling studies (i.e., certainty associated with model outputs).

Study design and setting: Expert consultations and an international multidisciplinary workshop informed development of a conceptual approach to assessing the certainty of evidence from models within the context of systematic reviews, health technology assessments, and health care decisions. The discussions also clarified selected concepts and terminology used in the GRADE approach and by the modeling community. Feedback from experts in a broad range of modeling and health care disciplines addressed the content validity of the approach.

Results: Workshop participants agreed that the domains determining the certainty of evidence previously identified in the GRADE approach (risk of bias, indirectness, inconsistency, imprecision, reporting bias, magnitude of an effect, dose-response relation, and the direction of residual confounding) also apply when assessing the certainty of evidence from models. The assessment depends on the nature of model inputs and the model itself and on whether one is evaluating evidence from a single model or multiple models. We propose a framework for selecting the best available evidence from models: 1) developing de novo, a model specific to the situation of interest, 2) identifying an existing model, the outputs of which provide the highest certainty evidence for the situation of interest, either "off-the-shelf" or after adaptation, and 3) using outputs from multiple models. We also present a summary of preferred terminology to facilitate communication among modeling and health care disciplines.

Conclusion: This conceptual GRADE approach provides a framework for using evidence from models in health decision-making and the assessment of certainty of evidence from a model or models. The GRADE Working Group and the modeling community are currently developing the detailed methods and related guidance for assessing specific domains determining the certainty of evidence from models across health care-related disciplines (e.g., therapeutic decision-making, toxicology, environmental health, and health economics).

Keywords: Certainty of evidence; GRADE; Guidelines; Health care Decision making; Mathematical models; Modelling studies.

PubMed Disclaimer

Figures

**Fig. 1.**
The general approach to using modeled evidence and assessing its certainty in health-related disciplines.

See this image and copyright information in PMC

Cited by

Costs and cost-effectiveness of treatment setting for children with wasting, oedema and growth failure/faltering: A systematic review.
Mdege ND, Masuku SD, Musakwa N, Chisala M, Tingum EN, Boachie MK, Shokraneh F. Mdege ND, et al. PLOS Glob Public Health. 2023 Nov 8;3(11):e0002551. doi: 10.1371/journal.pgph.0002551. eCollection 2023. PLOS Glob Public Health. 2023. PMID: 37939029 Free PMC article.
Does Reiki Benefit Mental Health Symptoms Above Placebo?
Zadro S, Stapleton P. Zadro S, et al. Front Psychol. 2022 Jul 12;13:897312. doi: 10.3389/fpsyg.2022.897312. eCollection 2022. Front Psychol. 2022. PMID: 35911042 Free PMC article.
Decoding the Microbiome's Influence on Rheumatoid Arthritis.
Coradduzza D, Bo M, Congiargiu A, Azara E, De Miglio MR, Erre GL, Carru C. Coradduzza D, et al. Microorganisms. 2023 Aug 28;11(9):2170. doi: 10.3390/microorganisms11092170. Microorganisms. 2023. PMID: 37764014 Free PMC article. Review.
Is levator ani avulsion a risk factor for prolapse recurrence? A systematic review and meta-analysis.
Yeung E, Malacova E, Maher C. Yeung E, et al. Int Urogynecol J. 2022 Jul;33(7):1813-1826. doi: 10.1007/s00192-022-05217-2. Epub 2022 May 10. Int Urogynecol J. 2022. PMID: 35538253 Free PMC article.
Identifying best modelling practices for tobacco control policy simulations: a systematic review and a novel quality assessment framework.
Huang V, Head A, Hyseni L, O'Flaherty M, Buchan I, Capewell S, Kypridemos C. Huang V, et al. Tob Control. 2023 Sep;32(5):589-598. doi: 10.1136/tobaccocontrol-2021-056825. Epub 2022 Jan 11. Tob Control. 2023. PMID: 35017262 Free PMC article.

See all "Cited by" articles

References

1. Oreskes N. The role of quantitative models in science. In: Canham CD, Cole JJ, Lauenroth WK, editors. Models in ecosystem science. Princeton, NJ: Princeton University Press; 2003:13–31.
1. Frigg R, Hartmann S. Models in science. In: Zalta EN, editor. The Stanford Encyclopedia of Philosophy. Stanford, CA: Spring 2017 Edition; 2017.
1. Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schunemann HJ, et al. What is “quality of evidence” and why is it important to clinicians? BMJ 2008;336:995–8. - PMC - PubMed
1. Oreskes N. Evaluation (not validation) of quantitative models. Environ Health Perspect 1998;106(Suppl 6):1453–60. - PMC - PubMed
1. Briggs AH, Weinstein MC, Fenwick EA, Karnon J, Sculpher MJ, Paltiel AD, et al. Model parameter estimation and uncertainty: a report of the ISPOR-SMDM modeling good research practices task force–6. Value Health 2012;15:835–42. - PubMed

Publication types

Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions

Grants and funding

LinkOut - more resources

Full Text Sources
Other Literature Sources
- scite Smart Citations

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

GRADE Guidelines 30: the GRADE approach to assessing the certainty of modeled evidence-An overview in the context of health decision-making

Affiliations

GRADE Guidelines 30: the GRADE approach to assessing the certainty of modeled evidence-An overview in the context of health decision-making

Authors

Affiliations

Abstract

Figures

Similar articles

Cited by

References

Publication types

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources

Other Literature Sources

Abstract

Figures

Similar articles

Cited by

References

Publication types

MeSH terms

Related information

Grants and funding

LinkOut - more resources

Full Text Sources

Other Literature Sources