Practice Guideline

. 2017 Jul:87:4-13.

doi: 10.1016/j.jclinepi.2017.05.006. Epub 2017 May 18.

The GRADE Working Group clarifies the construct of certainty of evidence

Monica Hultcrantz¹, David Rind², Elie A Akl³, Shaun Treweek⁴, Reem A Mustafa⁵, Alfonso Iorio⁶, Brian S Alper⁷, Joerg J Meerpohl⁸, M Hassan Murad⁹, Mohammed T Ansari¹⁰, Srinivasa Vittal Katikireddi¹¹, Pernilla Östlund¹², Sofia Tranæus¹³, Robin Christensen¹⁴, Gerald Gartlehner¹⁵, Jan Brozek⁶, Ariel Izcovich¹⁶, Holger Schünemann⁶, Gordon Guyatt⁶

Affiliations

¹ Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU), S:t Eriksgatan 117, SE-102 33, Stockholm, Sweden; Department of Learning, Informatics, Management and Ethics, Karolinska Institutet, Tomtebodav. 18 A, SE-171 77, Stockholm, Sweden. Electronic address: monica.hultcrantz@sbu.se.
² Institute for Clinical and Economic Review, Two Liberty Square, 9(th) Floor, Boston, MA 02109, USA; Harvard Medical School Beth Israel Deaconess Medical Center, 330 Brookline Avenue Boston, MA 02215, USA.
³ Department of Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Department of Internal Medicine, American University of Beirut Medical Center, P.O. Box 11-0236, Riad-El-Solh, Beirut 1107 2020, Lebanon.
⁴ Health Services Research Unit, University of Aberdeen, 3rd Floor, Health Sciences Building, Foresterhill, Aberdeen, AB25 2ZD, UK.
⁵ Department of Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Department of Medicine, Division of Nephrology and Hypertension, University of Kansas Medical Center, 3901 Rainbow Blvd, MS3002, Kansas City, KS 66160, USA.
⁶ Department of Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Department of Medicine, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada.
⁷ EBSCO Health, EBSCO Publishing, 10 Estes Street, Ipswich, MA, 01938 USA; Department of Family and Community Medicine, School of Medicine, University of Missouri-Columbia, One Hospital Drive, Columbia, MO 65212, USA.
⁸ Cochrane Germany, Medical Center - University of Freiburg, Faculty of Medicine, University of Freiburg, Breisacher Strasse 153,79110 Freiburg, Germany; Centre de Recherche Épidémiologie et Statistique Sorbonne Paris Cité - U1153, Inserm / Université Paris Descartes; Cochrane France; Hôpital Hôtel-Dieu, 1 place du Parvis Notre Dame, 75181 Paris Cedex 04, France.
⁹ Mayo Clinic Evidence-based Practice Center, Mayo Clinic, 200 1st Street SW, Rochester, MN 55905, USA.
¹⁰ School of Epidemiology, Public Health and Preventive Medicine, Faculty of Medicine, University of Ottawa, 600 Peter Morand Crescent, Ottawa, Ontario K1G 5Z3, Canada.
¹¹ MRC/CSO Social & Public Health Sciences Unit, University of Glasgow, 200 Renfield Street, Glasgow, G2 3QB, UK.
¹² Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU), S:t Eriksgatan 117, SE-102 33, Stockholm, Sweden; Health Technology Assessment - Odontology (HTA-O), Faculty of Odontology, Malmö University, Carl Gustafs Väg 34, SE-214 21, Malmö, Sweden.
¹³ Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU), S:t Eriksgatan 117, SE-102 33, Stockholm, Sweden; Health Technology Assessment - Odontology (HTA-O), Faculty of Odontology, Malmö University, Carl Gustafs Väg 34, SE-214 21, Malmö, Sweden; Division of Cariology and Endodontology, Department of Dental Medicine, Karolinska institutet, Alfred Nobels Allé 8, SE-141 52, Huddinge, Sweden.
¹⁴ Musculoskeletal Statistics Unit, The Parker Institute, Bispebjerg and Frederiksberg Hospital, Nordre Fasanvej 57, DK-2000 Copenhagen F, Denmark.
¹⁵ Danube University, Department for Evidence-based Medicine and Clinical Epidemiology, Dr. Karl Dorrekstrasse 30, 3500 Krems, Austria; RTI International, Research Triangle Park, 3040 East Cornwallis Rd, Research Triangle Park 27709, NC, USA.
¹⁶ Internal Medicine Service, German Hospital, Pueyrredón 1640, Buenos Aires C1118AAT, Argentina.

PMID: 28529184
PMCID: PMC6542664
DOI: 10.1016/j.jclinepi.2017.05.006

Practice Guideline

The GRADE Working Group clarifies the construct of certainty of evidence

Monica Hultcrantz et al. J Clin Epidemiol. 2017 Jul.

. 2017 Jul:87:4-13.

doi: 10.1016/j.jclinepi.2017.05.006. Epub 2017 May 18.

Authors

Affiliations

¹ Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU), S:t Eriksgatan 117, SE-102 33, Stockholm, Sweden; Department of Learning, Informatics, Management and Ethics, Karolinska Institutet, Tomtebodav. 18 A, SE-171 77, Stockholm, Sweden. Electronic address: monica.hultcrantz@sbu.se.
² Institute for Clinical and Economic Review, Two Liberty Square, 9(th) Floor, Boston, MA 02109, USA; Harvard Medical School Beth Israel Deaconess Medical Center, 330 Brookline Avenue Boston, MA 02215, USA.
³ Department of Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Department of Internal Medicine, American University of Beirut Medical Center, P.O. Box 11-0236, Riad-El-Solh, Beirut 1107 2020, Lebanon.
⁴ Health Services Research Unit, University of Aberdeen, 3rd Floor, Health Sciences Building, Foresterhill, Aberdeen, AB25 2ZD, UK.
⁵ Department of Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Department of Medicine, Division of Nephrology and Hypertension, University of Kansas Medical Center, 3901 Rainbow Blvd, MS3002, Kansas City, KS 66160, USA.
⁶ Department of Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Department of Medicine, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada.
⁷ EBSCO Health, EBSCO Publishing, 10 Estes Street, Ipswich, MA, 01938 USA; Department of Family and Community Medicine, School of Medicine, University of Missouri-Columbia, One Hospital Drive, Columbia, MO 65212, USA.
⁸ Cochrane Germany, Medical Center - University of Freiburg, Faculty of Medicine, University of Freiburg, Breisacher Strasse 153,79110 Freiburg, Germany; Centre de Recherche Épidémiologie et Statistique Sorbonne Paris Cité - U1153, Inserm / Université Paris Descartes; Cochrane France; Hôpital Hôtel-Dieu, 1 place du Parvis Notre Dame, 75181 Paris Cedex 04, France.
⁹ Mayo Clinic Evidence-based Practice Center, Mayo Clinic, 200 1st Street SW, Rochester, MN 55905, USA.
¹⁰ School of Epidemiology, Public Health and Preventive Medicine, Faculty of Medicine, University of Ottawa, 600 Peter Morand Crescent, Ottawa, Ontario K1G 5Z3, Canada.
¹¹ MRC/CSO Social & Public Health Sciences Unit, University of Glasgow, 200 Renfield Street, Glasgow, G2 3QB, UK.
¹² Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU), S:t Eriksgatan 117, SE-102 33, Stockholm, Sweden; Health Technology Assessment - Odontology (HTA-O), Faculty of Odontology, Malmö University, Carl Gustafs Väg 34, SE-214 21, Malmö, Sweden.
¹³ Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU), S:t Eriksgatan 117, SE-102 33, Stockholm, Sweden; Health Technology Assessment - Odontology (HTA-O), Faculty of Odontology, Malmö University, Carl Gustafs Väg 34, SE-214 21, Malmö, Sweden; Division of Cariology and Endodontology, Department of Dental Medicine, Karolinska institutet, Alfred Nobels Allé 8, SE-141 52, Huddinge, Sweden.
¹⁴ Musculoskeletal Statistics Unit, The Parker Institute, Bispebjerg and Frederiksberg Hospital, Nordre Fasanvej 57, DK-2000 Copenhagen F, Denmark.
¹⁵ Danube University, Department for Evidence-based Medicine and Clinical Epidemiology, Dr. Karl Dorrekstrasse 30, 3500 Krems, Austria; RTI International, Research Triangle Park, 3040 East Cornwallis Rd, Research Triangle Park 27709, NC, USA.
¹⁶ Internal Medicine Service, German Hospital, Pueyrredón 1640, Buenos Aires C1118AAT, Argentina.

PMID: 28529184
PMCID: PMC6542664
DOI: 10.1016/j.jclinepi.2017.05.006

Abstract

Objective: To clarify the grading of recommendations assessment, development and evaluation (GRADE) definition of certainty of evidence and suggest possible approaches to rating certainty of the evidence for systematic reviews, health technology assessments, and guidelines.

Study design and setting: This work was carried out by a project group within the GRADE Working Group, through brainstorming and iterative refinement of ideas, using input from workshops, presentations, and discussions at GRADE Working Group meetings to produce this document, which constitutes official GRADE guidance.

Results: Certainty of evidence is best considered as the certainty that a true effect lies on one side of a specified threshold or within a chosen range. We define possible approaches for choosing threshold or range. For guidelines, what we call a fully contextualized approach requires simultaneously considering all critical outcomes and their relative value. Less-contextualized approaches, more appropriate for systematic reviews and health technology assessments, include using specified ranges of magnitude of effect, for example, ranges of what we might consider no effect, trivial, small, moderate, or large effects.

Conclusion: It is desirable for systematic review authors, guideline panelists, and health technology assessors to specify the threshold or ranges they are using when rating the certainty in evidence.

Keywords: Certainty of evidence; GRADE; Guidelines; Health technology assessment; Systematic reviews; Thresholds.

PubMed Disclaimer

Figures

**Figure 1. Rating certainty that the true effect lies in a particular range: an illustration from previous GRADE writings (2)**

See this image and copyright information in PMC

Comment in

The construct of certainty of evidence has not been disseminated to systematic reviews and clinical practice guidelines; response to 'The GRADE Working Group' et al.
Taito S, Tsujimoto Y, Kataoka Y. Taito S, et al. J Clin Epidemiol. 2022 Jul;147:171-172. doi: 10.1016/j.jclinepi.2022.03.013. Epub 2022 Mar 24. J Clin Epidemiol. 2022. PMID: 35341947 No abstract available.

References

1. Balshem H, Helfand M, Schunemann HJ, Oxman AD, Kunz R, Brozek J, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol. 2011;64(4):401–6. - PubMed
1. Guyatt GH, Oxman AD, Kunz R, Brozek J, Alonso-Coello P, Rind D, et al. GRADE guidelines 6. Rating the quality of evidence--imprecision. J Clin Epidemiol. 2011;64(12):1283–93. - PubMed
1. Welch VA, Akl EA, Pottie K, Ansari MT, Briel M, Christensen R, et al. GRADE Equity Guidelines 3: Health equity considerations in rating the certainty of synthesized evidence. J Clin Epidemiol. 2017 - PMC - PubMed
1. Schunemann HJ. Interpreting GRADE's levels of certainty or quality of the evidence: GRADE for statisticians, considering review information size or less emphasis on imprecision? J Clin Epidemiol. 2016;75:6–15. - PubMed
1. Schunemann HJ. Guidelines 2.0: do no net harm-the future of practice guideline development in asthma and other diseases. Curr Allergy Asthma Rep. 2011;11(3):261–8. - PubMed

Publication types

Actions

MeSH terms

Actions
Actions
Actions
Actions

Grants and funding

LinkOut - more resources

Full Text Sources
Other Literature Sources
- scite Smart Citations
Medical
- ClinicalTrials.gov
Miscellaneous
- NCI CPTAC Assay Portal

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

The GRADE Working Group clarifies the construct of certainty of evidence

Affiliations

The GRADE Working Group clarifies the construct of certainty of evidence

Authors

Affiliations

Abstract

Figures

Comment in

References

Publication types

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources

Other Literature Sources

Medical

Miscellaneous