Inclusion of nonrandomized studies in Cochrane systematic reviews was found to be in need of improvement

Abstract

Objectives: Nonrandomized studies (NRSs) are considered to provide less reliable evidence for intervention effects. However, these are included in Cochrane reviews, despite discouragement. There has been no evaluation of when and how these designs are used. Therefore, we conducted an overview of current practice.

Study design and setting: We included all Cochrane reviews that considered NRS, conducting inclusions and data extraction in duplicate.

Results: Of the included 202 reviews, 114 (56%) did not cite a reason for including NRS. The reasons were divided into two major categories: NRS were included because randomized controlled trials (RCTs) are wanted (N = 81, 92%) but not feasible, lacking, or insufficient alone or because RCTs are not needed (N = 7, 8%). A range of designs were included with controlled before-after studies as the most common. Most interventions were nonpharmaceutical and the settings nonmedical. For risk of bias assessment, Cochrane Effective Practice and Organisation of Care Group's checklists were used by most reviewers (38%), whereas others used a variety of checklists and self-constructed tools.

Conclusion: Most Cochrane reviews do not justify including NRS. When they do, most are not in line with Cochrane recommendations. Risk of bias assessment varies across reviews and needs improvement.

Keywords: Comparative effectiveness research; Epidemiologic bias; Intervention studies; Meta-analysis; Review; Systematic reviews.