Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 May 23:2022:406-413.
eCollection 2022.

Testing a filtering strategy for systematic reviews: evaluating work savings and recall

Randi Proescholdt  1 , Tzu-Kun Hsiao  1 , Jodi Schneider  1 , Aaron M Cohen  2 , Marian S McDonagh  2 , Neil R Smalheiser  3
Affiliations

Testing a filtering strategy for systematic reviews: evaluating work savings and recall

Randi Proescholdt et al. AMIA Jt Summits Transl Sci Proc. .

Abstract

Systematic reviews are extremely time-consuming. The goal of this work is to assess work savings and recall for a publication type filtering strategy that uses the output of two machine learning models, Multi-Tagger and web RCT Tagger, applied retrospectively to 10 systematic reviews on drug effectiveness. Our filtering strategy resulted in mean work savings of 33.6% and recall of 98.3%. Of 363 articles finally included in any of the systematic reviews, 7 were filtered out by our strategy, but 1 "error" was actually an article using a publication type that the SR team had not pre-specified as relevant for inclusion. Our analysis suggests that automated publication type filtering can potentially provide substantial work savings with minimal loss of included articles. Publication type filtering should be personalized for each systematic review and might be combined with other filtering or ranking methods to provide additional work savings for manual triage.

PubMed Disclaimer

Similar articles

See all similar articles

Cited by

References

    1. Borah R, Brown AW, Capers PL, Kaiser KA. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open [Internet] 2017;7(2):e012545. Feb 1 [cited 2019 Jun 4]. Available from: http://doi.org/10.1136/bmjopen-2016-012545. - PMC - PubMed
    1. Michelson M, Reuter K. The significant cost of systematic reviews and meta-analyses: A call for greater involvement of machine learning to assess the promise of clinical trials. Contemp Clin Trials Commun [Internet] 2019;16:100443. Dec 1 [cited 2021 Aug 2]. Available from: http://doi.org/10.1016/j.conctc.2019.100443. - PMC - PubMed
    1. Hamel C, Kelly SE, Thavorn K, Rice DB, Wells GA, Hutton B. An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes. BMC Med Res Methodol [Internet] 2020;20(1):256. Oct 15 [cited 2020 Nov 10]. Available from: https://doi.org/10.1186/s12874-020-01129-1. - PMC - PubMed
    1. Tsou AY, Treadwell JR, Erinoff E, Schoelles K. Machine learning for screening prioritization in systematic reviews: comparative performance of Abstrackr and EPPI-Reviewer. Syst Rev [Internet] 2020;9(1):73. Apr 2 [cited 2021 Jan 2]. Available from: https://doi.org/10.1186/s13643-020-01324-7. - PMC - PubMed
    1. Giummarra MJ, Lau G, Gabbe BJ. Evaluation of text mining to reduce screening workload for injury-focused systematic reviews. Inj Prev J Int Soc Child Adolesc Inj Prev [Internet]. 2019 Aug 26; Available from: http://doi.org/10.1136/injuryprev-2019-043247. - PubMed

LinkOut - more resources