A framework for synthesizing intervention evidence from multiple sources into a single certainty of evidence rating: Methodological developments from a US National Academies of Sciences, Engineering, and Medicine Committee

Abstract

Despite research investment and a growing body of diverse evidence there has been no comprehensive review and grading of evidence for public health emergency preparedness and response practices comparable to those in medicine and other public health fields. The National Academies of Sciences, Engineering, and Medicine convened an ad hoc committee to develop and use methods for grading and synthesizing diverse types of evidence to create a single certainty of intervention-related evidence to support recommendations for Public Health Emergency Preparedness and Response Research. A 13-step consensus building method was used. Experts were first canvassed in public meetings, and a comprehensive review of existing methods was undertaken. Although aspects of existing review methodologies and evidence grading systems were relevant, none adequately covered all requirements for this specific context. Starting with a desire to synthesize diverse sources of evidence not usually included in systematic reviews and using GRADE for assessing certainty and confidence in quantitative and qualitative evidence as the foundation, we developed a mixed-methods synthesis review and grading methodology that drew on (and in some cases adapted) those elements of existing frameworks and methods that were most applicable. Four topics were selected as test cases. The process was operationalized with a suite of method-specific reviews of diverse evidence types for each topic. Further consensus building was undertaken through stakeholder engagement and feedback The NASEM committee's GRADE adaption for mixed-methods reviews will further evolve over time and has yet to be endorsed by the GRADE working group.

Keywords: GRADE; GRADE CERQυal; evidence synthesis; mixed-methods; systematic review.

FIGURE 1

Classification and consolidation of studies into methodological streams. Abbreviations: AAR, after action report; CASP, Critical Appraisal Skills Programme; certainty of evidence, certainty of the evidence; GRADE, Grading of Recommendations Assessment, Development and Evaluation; HSDL, Homeland Security Digital Library; RoB, risk of bias.*Risk of bias assessment tools were developed by adapting existing tools and/or published methods

FIGURE 2

Framework for integrating evidence to inform recommendation and guidance development for PHEPR practices. The framework depicts two interconnected pathways for evaluating evidence for PHEPR practices. The lefthand panel (blue) shows our process for integrating evidence from quantitative impact studies with other evidence that may inform what works to determine the certainty of the evidence (certainty of evidence) of effectiveness for a given outcome. The certainty of evidence (for all relevant outcomes) feeds into the righthand panel (white), which shows the pathways for integrating diverse evidence for various elements (evidence to decision elements) that, along with context considerations, may inform the formulation of evidence‐based practice recommendations and implementation guidance. In cases in which the review is focused on implementation and not on determining the effectiveness of a practice, it is possible to follow the pathway depicted in the righthand panel without assessing the certainty of evidence as shown in the left hand panel [Colour figure can be viewed at wileyonlinelibrary.com]