Evidence that nonsignificant results are sometimes preferred: Reverse P-hacking or selective reporting?

Abstract

There is increased concern about poor scientific practices arising from an excessive focus on P-values. Two particularly worrisome practices are selective reporting of significant results and 'P-hacking'. The latter is the manipulation of data collection, usage, or analyses to obtain statistically significant outcomes. Here, we introduce the novel, to our knowledge, concepts of selective reporting of nonsignificant results and 'reverse P-hacking' whereby researchers ensure that tests produce a nonsignificant result. We test whether these practices occur in experiments in which researchers randomly assign subjects to treatment and control groups to minimise differences in confounding variables that might affect the focal outcome. By chance alone, 5% of tests for a group difference in confounding variables should yield a significant result (P < 0.05). If researchers less often report significant findings and/or reverse P-hack to avoid significant outcomes that undermine the ethos that experimental and control groups only differ with respect to actively manipulated variables, we expect significant results from tests for group differences to be under-represented in the literature. We surveyed the behavioural ecology literature and found significantly more nonsignificant P-values reported for tests of group differences in potentially confounding variables than the expected 95% (P = 0.005; N = 250 studies). This novel, to our knowledge, publication bias could result from selective reporting of nonsignificant results and/or from reverse P-hacking. We encourage others to test for a bias toward publishing nonsignificant results in the equivalent context in their own research discipline.

Fig 1. Frequency distribution of numerical P-values (P-curve).

Two hypothetical examples of P-curves show the expected distribution if (a) selection bias is present or (b) certain types of reverse P-hacking occur (see [12]). Data collected from articles testing the effect of confounding variables between treatment(s) and control groups are shown in (c). In (c), darker red points represent a higher frequency of P-values/bin among the randomised datasets. The horizontal red line denotes the theoretical even distribution across 20 bins (188 articles in 20 bins = 9.4 articles/bin). The grey rectangles emphasize intervals of interest in the two hypothetical scenarios. We only used P-values presented to at least two decimal places. The full data file of the 1,805 articles consulted and the 250 from which data were extracted and the R scripts used for the analysis and to generate the figures are available at https://figshare.com/s/f3bb7dfefdaa8976d3a1 or https://osf.io/au7yc/.