Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Comparative Study
. 2019 Jul;107(3):364-373.
doi: 10.5195/jmla.2019.622. Epub 2019 Jul 1.

Search results outliers among MEDLINE platforms

Christopher Sean Burns  1 , Robert M Shapiro 2nd  2 , Tyler Nix  3 , Jeffrey T Huber  4
Affiliations
Comparative Study

Search results outliers among MEDLINE platforms

Christopher Sean Burns et al. J Med Libr Assoc. 2019 Jul.

Abstract

Objective: Hypothetically, content in MEDLINE records is consistent across multiple platforms. Though platforms have different interfaces and requirements for query syntax, results should be similar when the syntax is controlled for across the platforms. The authors investigated how search result counts varied when searching records among five MEDLINE platforms.

Methods: We created 29 sets of search queries targeting various metadata fields and operators. Within search sets, we adapted 5 distinct, compatible queries to search 5 MEDLINE platforms (PubMed, ProQuest, EBSCOhost, Web of Science, and Ovid), totaling 145 final queries. The 5 queries were designed to be logically and semantically equivalent and were modified only to match platform syntax requirements. We analyzed the result counts and compared PubMed's MEDLINE result counts to result counts from the other platforms. We identified outliers by measuring the result count deviations using modified z-scores centered around PubMed's MEDLINE results.

Results: Web of Science and ProQuest searches were the most likely to deviate from the equivalent PubMed searches. EBSCOhost and Ovid were less likely to deviate from PubMed searches. Ovid's results were the most consistent with PubMed's but appeared to apply an indexing algorithm that resulted in lower retrieval sets among equivalent searches in PubMed. Web of Science exhibited problems with exploding or not exploding Medical Subject Headings (MeSH) terms.

Conclusion: Platform enhancements among interfaces affect record retrieval and challenge the expectation that MEDLINE platforms should, by default, be treated as MEDLINE. Substantial inconsistencies in search result counts, as demonstrated here, should raise concerns about the impact of platform-specific influences on search results.

PubMed Disclaimer

Figures

Figure 1
Figure 1
Total search result counts for each of the 29 search sets The four plots are organized by the magnitude of results.
Figure 2
Figure 2
Deviations per platform from PubMed’s MEDLINE, excluding outlier searches
Figure 3
Figure 3
Outlier search results in ProQuest and Web of Science Numbers represent modified z-scores. A score outside of +/−3.5 is considered an outlier.
See this image and copyright information in PMC

References

    1. Amrhein V, Korner-Nievergelt F, Roth T. The earth is flat (p>0.05): significance thresholds and the crisis of unreplicable research. Peer J. 2017 Jul 7;5:e3544. doi: 10.7717/peerj.3544. - DOI - PMC - PubMed
    1. Baker M. 1,500 scientists lift the lid on reproducibility. Nature. 2016 May 26;533(7604):452–4. doi: 10.1038/533452a. - DOI - PubMed
    1. Open Science Collaboration. Estimating the reproducibility of psychological science. Science. 2015 Aug 28;349(6251):aac4716. doi: 10.1126/science.aac4716. doi: 10.1126/science.aac4716. - DOI - DOI - PubMed
    1. Moher D, Liberati A, Tetzlaff J, Altman DG, Group TP. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement. PLOS Med. 2009 Jul 21;6(7):e1000097. doi: 10.1371/journal.pmed.1000097. doi: 10.1371/journal.pmed.1000097. - DOI - DOI - PMC - PubMed
    1. Cochrane. Cochrane handbook for systematic reviews of interventions [Internet] Cochrane; [cited 6 May 2019]. < https://training.cochrane.org/handbook>.

Publication types

MeSH terms

LinkOut - more resources