Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 Mar;11(2):181-217.
doi: 10.1002/jrsm.1378. Epub 2020 Jan 28.

Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources

Michael Gusenbauer  1 , Neal R Haddaway  2   3
Affiliations

Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources

Michael Gusenbauer et al. Res Synth Methods. 2020 Mar.

Abstract

Rigorous evidence identification is essential for systematic reviews and meta-analyses (evidence syntheses) because the sample selection of relevant studies determines a review's outcome, validity, and explanatory power. Yet, the search systems allowing access to this evidence provide varying levels of precision, recall, and reproducibility and also demand different levels of effort. To date, it remains unclear which search systems are most appropriate for evidence synthesis and why. Advice on which search engines and bibliographic databases to choose for systematic searches is limited and lacking systematic, empirical performance assessments. This study investigates and compares the systematic search qualities of 28 widely used academic search systems, including Google Scholar, PubMed, and Web of Science. A novel, query-based method tests how well users are able to interact and retrieve records with each system. The study is the first to show the extent to which search systems can effectively and efficiently perform (Boolean) searches with regards to precision, recall, and reproducibility. We found substantial differences in the performance of search systems, meaning that their usability in systematic searches varies. Indeed, only half of the search systems analyzed and only a few Open Access databases can be recommended for evidence syntheses without adding substantial caveats. Particularly, our findings demonstrate why Google Scholar is inappropriate as principal search system. We call for database owners to recognize the requirements of evidence synthesis and for academic journals to reassess quality requirements for systematic reviews. Our findings aim to support researchers in conducting better searches for better evidence synthesis.

Keywords: academic search systems; discovery; evaluation; information retrieval; systematic review; systematic search.

PubMed Disclaimer

Conflict of interest statement

The author reported no conflict of interest.

References

    1. Price DJS. Little Science, Big Science. New York: Columbia Univ. Press; 1963. Columbia paperback.
    1. Larsen PO, von Ins M. The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index. Scientometrics. 2010;84(3):575‐603. 10.1007/s11192-010-0202-z - DOI - PMC - PubMed
    1. Eden D. From the editors: replication, meta‐analysis, scientific progress, and AMJ's publication policy. AMJ. 2002;45(5):841‐846. 10.5465/AMJ.2002.7718946 - DOI
    1. Naisbitt J, Aburdene P. Megatrends 2000: Ten New Directions for the 1990's. 1st ed. New York: Morrow; 1990.
    1. Cooper HM. Research Synthesis and Meta‐analysis: A Step‐by‐Step Approach. Applied social research methods series Fifth ed. 2 Los Angeles: SAGE; 2017.

LinkOut - more resources