Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2014 Oct 1;16(10):e223.
doi: 10.2196/jmir.3369.

Automatic evidence retrieval for systematic reviews

Miew Keen Choong  1 , Filippo Galgani, Adam G Dunn, Guy Tsafnat
Affiliations

Automatic evidence retrieval for systematic reviews

Miew Keen Choong et al. J Med Internet Res. .

Abstract

Background: Snowballing involves recursively pursuing relevant references cited in the retrieved literature and adding them to the search results. Snowballing is an alternative approach to discover additional evidence that was not retrieved through conventional search. Snowballing's effectiveness makes it best practice in systematic reviews despite being time-consuming and tedious.

Objective: Our goal was to evaluate an automatic method for citation snowballing's capacity to identify and retrieve the full text and/or abstracts of cited articles.

Methods: Using 20 review articles that contained 949 citations to journal or conference articles, we manually searched Microsoft Academic Search (MAS) and identified 78.0% (740/949) of the cited articles that were present in the database. We compared the performance of the automatic citation snowballing method against the results of this manual search, measuring precision, recall, and F1 score.

Results: The automatic method was able to correctly identify 633 (as proportion of included citations: recall=66.7%, F1 score=79.3%; as proportion of citations in MAS: recall=85.5%, F1 score=91.2%) of citations with high precision (97.7%), and retrieved the full text or abstract for 490 (recall=82.9%, precision=92.1%, F1 score=87.3%) of the 633 correctly retrieved citations.

Conclusions: The proposed method for automatic citation snowballing is accurate and is capable of obtaining the full texts or abstracts for a substantial proportion of the scholarly citations in review articles. By automating the process of citation snowballing, it may be possible to reduce the time and effort of common evidence surveillance tasks such as keeping trial registries up to date and conducting systematic reviews.

Keywords: evidence-based medicine; information storage and retrieval; medical informatics.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest: None declared.

Figures

Figure 1
Figure 1
Microsoft Academic Search (MAS).
Figure 2
Figure 2
Summary of the evaluation results (from 20 reviews with 949 scholarly citations, MAS included 740 citations, 633 citations were found, and 490 full texts and abstracts were downloaded automatically).
See this image and copyright information in PMC

References

    1. Ceri S, Bozzon A, Brambilla M, Della Valle E, Fraternali P, Quarteroni S. The information retrieval process. In: Carey MJ, Ceri S, editors. Web Information Retrieval. Heidelberg: Springer; 2013. pp. 13–26.
    1. Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ. 2005 Nov 5;331(7524):1064–5. doi: 10.1136/bmj.38636.593461.68. http://europepmc.org/abstract/MED/16230312 - DOI - PMC - PubMed
    1. Ramer SL. Site-ation pearl growing: methods and librarianship history and theory. J Med Libr Assoc. 2005 Jul;93(3):397–400. http://europepmc.org/abstract/MED/16059431 - PMC - PubMed
    1. Search Strategies and Heuristics for Searching Bibliographic and Full Text Databases. [2014-08-12]. https://dspace.creighton.edu/xmlui/bitstream/handle/10504/5903/Searching....
    1. Robinson KA, Dunn AG, Tsafnat G, Glasziou P. Citation networks of related trials are often disconnected: implications for bidirectional citation searches. J Clin Epidemiol. 2014 Jul;67(7):793–9. doi: 10.1016/j.jclinepi.2013.11.015. - DOI - PubMed

Publication types

LinkOut - more resources