Sensitivity analysis for publication bias in meta-analyses

Abstract

We propose sensitivity analyses for publication bias in meta-analyses. We consider a publication process such that 'statistically significant' results are more likely to be published than negative or "non-significant" results by an unknown ratio, η. Our proposed methods also accommodate some plausible forms of selection based on a study's standard error. Using inverse probability weighting and robust estimation that accommodates non-normal population effects, small meta-analyses, and clustering, we develop sensitivity analyses that enable statements such as 'For publication bias to shift the observed point estimate to the null, "significant" results would need to be at least 30 fold more likely to be published than negative or "non-significant" results'. Comparable statements can be made regarding shifting to a chosen non-null value or shifting the confidence interval. To aid interpretation, we describe empirical benchmarks for plausible values of η across disciplines. We show that a worst-case meta-analytic point estimate for maximal publication bias under the selection model can be obtained simply by conducting a standard meta-analysis of only the negative and 'non-significant' studies; this method sometimes indicates that no amount of such publication bias could 'explain away' the results. We illustrate the proposed methods by using real meta-analyses and provide an R package: PublicationBias.

Keywords: File drawer; Meta‐analysis; Publication bias; Sensitivity analysis.

Figure 1

(a) Standard contour‐enhanced funnel plot (Peters et al., 2008; Viechtbauer, 2010) versus (b) significance funnel plot for data generated with publication bias and with right‐skewed population effect sizes (η=10) (studies lying on the diagonal line have exactly p=0.05): , non‐affirmative; , affirmative; , robust independent point estimate within all studies; , robust independent point estimate within only the non‐affirmative studies

Figure 2

Significance funnel plot for the video games meta‐analysis of Anderson et al. (2010) (point estimates are on Fisher's z‐scale, the scale on which p‐values were computed; studies lying on the diagonal line have p=0.05): , non‐affirmative; , affirmative; , robust clustered estimate in non‐affirmative studies only; , robust clustered estimate in all studies

Figure 3

Corrected point estimates and CIs for the video games meta‐analysis of Anderson et al. (2010) as a function of η (, non‐null value q ^′; , worst‐case estimates): (a) common effect specification; (b) robust specifications (, CI assuming independence; , CI allowing for clustering)

Figure 4

Significance funnel plot for the cancer meta‐analysis of Li et al. (2018) (studies lying on the diagonal line have p=0.05): , non‐affirmative; , affirmative; , robust clustered estimate in non‐affirmative studies only; , robust clustered estimate in all studies

Figure 5

Corrected point estimates (HR) and CIs for the cancer meta‐analysis of Li et al. (2018) as a function of η (, non‐null value q ^′; , worst‐case estimates): (a) common effect specification; (b) robust specifications (, CI assuming independence; , CI allowing for clustering)

Figure 6

Significance funnel plot for the optimism meta‐analysis (studies lying on the diagonal line have p=0.05): , non‐affirmative; , affirmative; , robust clustered estimate in non‐affirmative studies only; , robust clustered estimate in all studies

Figure 7

Corrected point estimates (Pearson's r) and CIs for the optimism meta‐analysis as a function of η (, non‐null value q ^′; , worst‐case estimates): (a) common effect specification; (b) robust specifications ( CI assuming independence; , CI allowing for clustering)

Figure 8

Median number of published studies across all simulation iterates (rows, numbers of studies in the underlying population before publication bias (5M ^*); columns, distributions of study level random effects and true mean μ): , not selected on standard error; , selected for small standard error; , common effect; , robust clustered; , robust independent

Figure 9

Median number of published non‐affirmative studies across all simulation iterates (rows, numbers of studies in the underlying population before publication bias (5M ^*); columns, distribution of study level random effects and true mean μ): , not selected on standard error; , selected for small standard error; , common effect; , robust clustered; , robust independent

Figure 10

Mean point estimate ${\widehat{\mu }}_{\eta }$ across simulation iterates (rows, number of studies in the underlying population before publication bias (5M ^*); columns, distribution of study level random effects and true mean μ): , μ; , not selected on standard error; , selected for small standard error; , common effect; , robust clustered; , robust independent