Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 May;15(3):500-511.
doi: 10.1002/jrsm.1703. Epub 2024 Feb 7.

Footprint of publication selection bias on meta-analyses in medicine, environmental sciences, psychology, and economics

František Bartoš  1   2 , Maximilian Maier  3 , Eric-Jan Wagenmakers  1 , Franziska Nippold  4 , Hristos Doucouliagos  5 , John P A Ioannidis  6   7   8   9   10 , Willem M Otte  11 , Martina Sladekova  12 , Teshome K Deresssa  13 , Stephan B Bruns  6   13   14 , Daniele Fanelli  15   16 , T D Stanley  5
Affiliations

Footprint of publication selection bias on meta-analyses in medicine, environmental sciences, psychology, and economics

František Bartoš et al. Res Synth Methods. 2024 May.

Abstract

Publication selection bias undermines the systematic accumulation of evidence. To assess the extent of this problem, we survey over 68,000 meta-analyses containing over 700,000 effect size estimates from medicine (67,386/597,699), environmental sciences (199/12,707), psychology (605/23,563), and economics (327/91,421). Our results indicate that meta-analyses in economics are the most severely contaminated by publication selection bias, closely followed by meta-analyses in environmental sciences and psychology, whereas meta-analyses in medicine are contaminated the least. After adjusting for publication selection bias, the median probability of the presence of an effect decreased from 99.9% to 29.7% in economics, from 98.9% to 55.7% in psychology, from 99.8% to 70.7% in environmental sciences, and from 38.0% to 29.7% in medicine. The median absolute effect sizes (in terms of standardized mean differences) decreased from d = 0.20 to d = 0.07 in economics, from d = 0.37 to d = 0.26 in psychology, from d = 0.62 to d = 0.43 in environmental sciences, and from d = 0.24 to d = 0.13 in medicine.

Keywords: Bayesian; RoBMA; effect sizes; evidence; meta‐analysis; model‐averaging; publication bias.

PubMed Disclaimer

References

REFERENCES

    1. Chavalarias D, Ioannidis JP. Science mapping analysis characterizes 235 biases in biomedical research. J Clin Epidemiol. 2010;63(11):1205‐1215.
    1. Dwan K, Altman DG, Arnaiz JA, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PloS One. 2008;3(8):e3081.
    1. Rosenthal R, Gaito J. Further evidence for the cliff effect in interpretation of levels of significance. Psychol Rep. 1964;15(2):570.
    1. Wicherts JM. The weak spots in contemporary science (and how to fix them). Animals. 2017;7(12):90‐119.
    1. Otte WM, Vinkers CH, Habets PC, IJzendoorn vDG, Tijdink JK. Analysis of 567,758 randomized controlled trials published over 30 years reveals trends in phrases used to discuss results that do not reach statistical significance. PLoS Biol. 2022;20(2):e3001562.

LinkOut - more resources