A large-scale analysis of the reasons given for excluding articles that are retrieved by literature search during systematic review

Abstract

Objective: A literature search to identify relevant studies is one of the first steps in performing a systematic review (SR) in support of evidence-based medicine. To maximize efficiency, the search must find practically all relevant studies and retrieve few that are irrelevant; however, this level of precision is seldom attained. Therefore, many articles must be manually examined for relevance. To better understand the limitations of current search tools as applied to SR, we characterized the most common reasons that papers retrieved by SR searches were excluded from the review.

Methods: The textual reasons given for retrieved but excluded articles were extracted from 6,743 SRs performed by 54 Cochrane Collaboration review groups. The frequencies of different exclusion reasons were analyzed, and we developed a taxonomy summarizing these reasons.

Results: Almost 65% of articles were excluded because the means of comparison were inappropriate. Of these, about 72% were due to the randomized controlled trial (RCT) design being required but not employed by the excluded study. Mismatching interventions and outcomes and incorrect population characteristics were also common reasons for exclusion.

Conclusions: Currently available search methods do not adequately address the most common exclusion reasons for systematic review, even those based primarily on study design.

Figure 1.

Tree bubble diagram of the relative frequency of exclusion reasons, by top level PICO category. In order to keep the diagram readable we have abbreviated the exclusion reasons listed in Table 2 by removing negation terms such as “not” and “wrong.”