Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2018 Dec 18;18(1):171.
doi: 10.1186/s12874-018-0625-4.

Study filters for non-randomized studies of interventions consistently lacked sensitivity upon external validation

Elke Hausner  1 , Maria-Inti Metzendorf  2 , Bernd Richter  2 , Fabian Lotz  3 , Siw Waffenschmidt  4
Affiliations

Study filters for non-randomized studies of interventions consistently lacked sensitivity upon external validation

Elke Hausner et al. BMC Med Res Methodol. .

Abstract

Background: Little evidence is available on searches for non-randomized studies (NRS) in bibliographic databases within the framework of systematic reviews. For instance, it is currently unclear whether, when searching for NRS, effective restriction of the search strategy to certain study types is possible. The following challenges need to be considered: 1) For non-randomized controlled trials (NRCTs): whether they can be identified by established filters for randomized controlled trials (RCTs). 2) For other NRS types (such as cohort studies): whether study filters exist for each study type and, if so, which performance measures they have. The aims of the present analysis were to identify and validate existing NRS filters in MEDLINE as well as to evaluate established RCT filters using a set of MEDLINE citations.

Methods: Our analysis is a retrospective analysis of study filters based on MEDLINE citations of NRS from Cochrane reviews. In a first step we identified existing NRS filters. For the generation of the reference set, we screened Cochrane reviews evaluating NRS, which covered a broad range of study types. The citations of the studies included in the Cochrane reviews were identified via the reviews' bibliographies and the corresponding PubMed identification numbers (PMIDs) were extracted from PubMed. Random samples comprising up to 200 citations (i.e. 200 PMIDs) each were created for each study type to generate the test sets.

Results: A total of 271 Cochrane reviews from 41 different Cochrane groups were eligible for data extraction. We identified 14 NRS filters published since 2001. The study filters generated between 660,000 and 9.5 million hits in MEDLINE. Most filters covered several study types. The reference set included 2890 publications classified as NRS for the generation of the test sets. Twelve test sets were generated (one for each study type), of which 8 included 200 citations each. None of the study filters achieved sufficient sensitivity (≥ 92%) for all of the study types targeted.

Conclusions: The performance of current NRS filters is insufficient for effective use in daily practice. It is therefore necessary to develop new strategies (e.g. new NRS filters in combination with other search techniques). The challenges related to NRS should be taken into account.

Keywords: Databases, bibliographic; Information storage and retrieval; Medline; Reproducibility of results; Sensitivity and specificity.

PubMed Disclaimer

Conflict of interest statement

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Figures

Fig. 1
Fig. 1
Flowchart for generation of the test sets
Fig. 2
Fig. 2
Type of intervention examined by Cochrane reviews in the reference set (according to Polus et al. [9])
See this image and copyright information in PMC

References

    1. Deeks JJ, Dinnes J, D'Amico R, Sowden AJ, Sakarovitch C, Song F, et al. Evaluating non-randomised intervention studies. Health Technol Assess. 2003;7(iii-x):1–173. - PubMed
    1. Jenkins M. Evaluation of methodological search filters: a review. Health Inf Libr J. 2004;21:148–163. doi: 10.1111/j.1471-1842.2004.00511.x. - DOI - PubMed
    1. Lefebvre C, Manheimer E, Glanville J. Searching for studies. In: Higgins JPT, green S, editors. Cochrane handbook for systematic reviews of interventions: version 5.1.0. Cochrane collaboration; 2011. http://handbook-5-1.cochrane.org/index.htm#chapter_6/6_searching_for_stu.... Accessed 30 Jan 2018.
    1. Health Information Research Unit. Hedges. 2013. http://hiru.mcmaster.ca/hiru/HIRU_Hedges_home.aspx. Accessed 11 July 2013.
    1. Reeves BC, Deeks JJ, Higgins JPT, Wells GA. Including non-randomized studies. In: Higgins JPT, green S, editors. Cochrane handbook for systematic reviews of interventions: version 5.1.0. Cochrane collaboration; 2011. http://handbook-5-1.cochrane.org/chapter_13/13_including_non_randomized_.... Accessed 30 Jan 2018.

Publication types

MeSH terms

LinkOut - more resources