Over half of clinical practice guidelines use non-systematic methods to inform recommendations: A methods study

Abstract

Introduction: Assessing the process used to synthesize the evidence in clinical practice guidelines enables users to determine the trustworthiness of the recommendations. Clinicians are increasingly dependent on guidelines to keep up with vast quantities of medical literature, and guidelines are followed to avoid malpractice suits. We aimed to assess whether systematic methods were used when synthesizing the evidence for guidelines; and to determine the type of review cited in support of recommendations.

Methods: Guidelines published in 2017 and 2018 were retrieved from the TRIP and Epistemonikos databases. We randomly sorted and sequentially screened clinical guidelines on all topics to select the first 50 that met our inclusion criteria. Our primary outcomes were the number of guidelines using either a systematic or non-systematic process to gather, assess, and synthesise evidence; and the numbers of recommendations within guidelines based on different types of evidence synthesis (systematic or non-systematic reviews). If a review was cited, we looked for evidence that it was critically appraised, and recorded which quality assessment tool was used. Finally, we examined the relation between the use of the GRADE approach, systematic review process, and type of funder.

Results: Of the 50 guidelines, 17 (34%) systematically synthesised the evidence to inform recommendations. These 17 guidelines clearly reported their objectives and eligibility criteria, conducted comprehensive search strategies, and assessed the quality of the studies. Of the 29/50 guidelines that included reviews, 6 (21%) assessed the risk of bias of the review. The quality of primary studies was reported in 30/50 (60%) guidelines.

Conclusions: High quality, systematic review products provide the best available evidence to inform guideline recommendations. Using non-systematic methods compromises the validity and reliability of the evidence used to inform guideline recommendations, leading to potentially misleading and untrustworthy results.

Fig 1. Procedures used to collect, assess, and synthesise evidence to inform guideline recommendations (i.e. systematic, non-systematic).

Clinical practice guidelines can use a non-systematic or systematic process to collect, assess, and synthesise evidence to inform guideline recommendations. Guideline developers can conduct a (i) literature review (using non-systematic methods), (ii) systematic review (using systematic methods with inclusion of all eligible study types [e.g. primary studies, systematic reviews, overviews]), or (iii) an overview of systematic reviews (using either systematic or non-systematic methods with inclusion and synthesis of systematic reviews). Using these three evidence synthesis products, developers of guidelines can include only primary studies, both primary studies and systematic reviews, only systematic reviews, and/or both systematic reviews, clinical practice guidelines, health technology assessment (HTAs), or overviews of systematic reviews. This figure was adapted from Lunny et al. [24].

Fig 2. Systematic or non-systematic process used by guidelines in their approach to evidence synthesis (n = 50).

To determine if a systematic process was used to gather, assess and synthesise evidence to inform recommendations, we used the following four criteria: (1) explicit statement of the questions or objectives reported in terms of PICOS (Populations, Interventions, Comparisons, Outcomes, and Study design) elements; (2) eligibility criteria reported for all included study designs; (3) a systematic search conducted (i.e. two or more databases searched); and (4) process reported for selecting/screening studies (e.g. number of authors, independent process). We considered these criteria to be the minimum that can be used by a guideline to reduce bias and limitations when gathering evidence to inform recommendations.