GRADE guidance 36: updates to GRADE's approach to addressing inconsistency

Abstract

Objectives: To update previous Grading of Recommendations Assessment, Development and Evaluation (GRADE) guidance by addressing inconsistencies and interpreting subgroup analyses.

Study design and setting: Using an iterative process, we consulted with members of the GRADE working group through multiple rounds of written feedback and discussions at GRADE working group meetings.

Results: The guidance complements previous guidance with clarification in two areas: (1) assessing inconsistency and (2) assessing the credibility of possible effect modifiers that might explain inconsistency. Specifically, the guidance clarifies that inconsistency refers to variability in results, not in study characteristics; that inconsistency assessment for binary outcomes requires consideration of both relative and absolute effects; how to decide between narrower and broader questions in systematic reviews and guidelines; that, with the same evidence, ratings of inconsistency may differ depending on the target of certainty rating; and how GRADE inconsistency ratings relate to a statistical measure of inconsistency I² depending on the context in which one views results. The second part of the guidance illustrates, based on a worked example, the use of the instrument to assess the credibility of effect modification analyses. The guidance explains the stepwise process of moving from a subgroup analysis to assessing the credibility of effect modification and, if found credible, to subgroup-specific effect estimates and GRADE certainty ratings.

Conclusion: This updated guidance addresses specific conceptual and practical issues that systematic review authors frequently face when considering the degree of inconsistency in estimates of treatment effects across studies.

Keywords: Certainty in evidence; Effect modification; GRADE; Heterogeneity; Inconsistency; Subgroup analysis; Systematic review.