Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2016 May 5;16(1):53.
doi: 10.1186/s12903-016-0208-x.

Significance bias: an empirical evaluation of the oral health literature

Edwin Kagereki  1 , Joseph Gakonyo  2 , Hazel Simila  2
Affiliations

Significance bias: an empirical evaluation of the oral health literature

Edwin Kagereki et al. BMC Oral Health. .

Abstract

Background: The tendency to selectively report "significant" statistical results (file-drawers effect) or run selective analyses to achieve "significant" results (data-dredging) has been observed in many scientific fields. Subsequently, statistically significant findings may be due to selective reporting rather than a true effect. The p-curve, a distribution of p-values from a set of studies, is used to study aspects of statistical evidence in a scientific field. The aim of this study was to assess publication bias and evidential value in oral health research.

Methods: This was a descriptive and exploratory study that analysed the p-values published in oral health literature. The National Library of Medicine catalogue was searched for journals published in English, indexed in PubMed and tagged with dentistry Medical Subject Headings (MeSH) words. Web scraping for abstracts published between 2004 and 2014 was done and all p-values extracted. A p-curve was generated from the p-values and used for analysis. Bayesian binomial analysis was used to test the proportion of the p-values on either side of the 0.05 threshold (test for publication bias) or the 0.025 threshold (test for evidential value). The tacit assumption was that significant p-values reported were the result of publication bias.

Results: The present study found the use of p-values in a total of 44,315 p-values published in 12,440 abstracts. Two percent of the p-values were inaccurately reported as zero or ≥1. The p-curve was right skewed, with an intriguing bi-modality. The distribution of the p-values is also unequal on either side of 0.025 and 0.045 of the p-curve.

Conclusions: This study found evidence of data-dredging, publication bias and errors in the dental literature. Although the present study was conducted on abstracts, the findings highlight a subject that should be researched in future studies that would consider the various factors that may influence p-values.

Keywords: Data-dredging; Evidential value; File drawer effect; P-value; Significance bias.

PubMed Disclaimer

Figures

Fig. 1
Fig. 1
Search strategy. The National Library of Medicine (NLM) catalogue was searched for journals published in English, indexed in PubMed and tagged with dentistry MeSH (Medical Subject Headings) words (MeSH Unique ID: D003813). Repeated entries and journals with missing volumes within the study period were excluded
Fig. 2
Fig. 2
The p-curve of the 44,315 p-values studied. The curve on the left a illustrates the overabundance of the p-values below the 0.05 threshold. The curve on the right b is a closer look at the p-values below the 0.05 threshold illustrating a bi-modal distribution of the p-values; one peak close to zero and the other close to the conventional significant threshold of 0.05
See this image and copyright information in PMC

References

    1. Alec C et al. Goodhart’s Law: Its Origins, Meaning and Implications for Monetary Policy. A Festschrift in honour of Charles Goodhart held on 15-16 November 2001 at the Bank of England.
    1. Fanelli D. Negative results are disappearing from most disciplines and countries. Scientometrics. 2012;90:891–904. doi: 10.1007/s11192-011-0494-7. - DOI
    1. Rosenthal R. The file drawer problem and tolerance for null results. Psychol Bull. 1979;86:638–641. doi: 10.1037/0033-2909.86.3.638. - DOI
    1. Sterne JA, Egger M, Smith GD. Systematic reviews in health care: investigating and dealing with publication and other biases in meta-analysis. BMJ. 2001;323:101. doi: 10.1136/bmj.323.7304.101. - DOI - PMC - PubMed
    1. Ferguson CJ, Heene M. A vast graveyard of undead theories: publication bias and psychological Science’s aversion to the null. Perspect Psychol Sci. 2012;7(6):555–561. doi: 10.1177/1745691612459059. - DOI - PubMed