Conducting Meta-Analyses Based on p Values: Reservations and Recommendations for Applying p-Uniform and p-Curve

Abstract

Because of overwhelming evidence of publication bias in psychology, techniques to correct meta-analytic estimates for such bias are greatly needed. The methodology on which the p-uniform and p-curve methods are based has great promise for providing accurate meta-analytic estimates in the presence of publication bias. However, in this article, we show that in some situations, p-curve behaves erratically, whereas p-uniform may yield implausible estimates of negative effect size. Moreover, we show that (and explain why) p-curve and p-uniform result in overestimation of effect size under moderate-to-large heterogeneity and may yield unpredictable bias when researchers employ p-hacking. We offer hands-on recommendations on applying and interpreting results of meta-analyses in general and p-uniform and p-curve in particular. Both methods as well as traditional methods are applied to a meta-analysis on the effect of weight on judgments of importance. We offer guidance for applying p-uniform or p-curve using R and a user-friendly web application for applying p-uniform.

Keywords: heterogeneity; meta-analysis; p-curve; p-hacking; p-uniform.

Fig. 1.

Probability-probability (P-P) plot for a meta-analysis of 20 studies with large heterogeneity.

Fig. 2.

Values for Kolmogorov-Smirnov (KS) test statistics in implementation of p-curve and p-uniform for the example with three observed effect sizes and p values close to .05. D stat = test statistics of KS test.

Fig. 3.

Effect size estimates in p-uniform and fixed-effect meta-analysis in case of four types of p-hacking. FE = fixed-effect; DV = dependent variable.

Fig. A1.

Computation of conditional p values for Effect 3 (q₃) for three effect sizes: δ = 0; δ = 0.5 (true effect size); and δ = 0.748 (estimate of fixed-effect meta-analysis). Critical value on Cohen’s d scale is denoted by d_cv and observed effect size is denoted by d_obs.

Fig. A2.

Observed conditional p values (solid black lines) and conditional p values under uniformity (dashed gray lines) for the example with three observed effect sizes. The three panels refer to the conditional p values for the p-uniform hypothesis test of no effect (δ = 0), p-uniform effect size estimate (δ = 0.5), and effect size obtained by fixed-effect (FE) meta-analysis (δ = 0.748).

Fig. A3.

Conditional p values as a function of true effect size (x axis) for each of the three observed effect sizes in the example. Effect sizes zero (δ = 0), true effect size (δ = 0.5), and estimated by fixed-effect meta-analysis (δ = 0.748) are indicated by vertical lines.