Pitfalls of using the risk ratio in meta-analysis

Abstract

For meta-analysis of studies that report outcomes as binomial proportions, the most popular measure of effect is the odds ratio (OR), usually analyzed as log(OR). Many meta-analyses use the risk ratio (RR) and its logarithm because of its simpler interpretation. Although log(OR) and log(RR) are both unbounded, use of log(RR) must ensure that estimates are compatible with study-level event rates in the interval (0, 1). These complications pose a particular challenge for random-effects models, both in applications and in generating data for simulations. As background, we review the conventional random-effects model and then binomial generalized linear mixed models (GLMMs) with the logit link function, which do not have these complications. We then focus on log-binomial models and explore implications of using them; theoretical calculations and simulation show evidence of biases. The main competitors to the binomial GLMMs use the beta-binomial (BB) distribution, either in BB regression or by maximizing a BB likelihood; a simulation produces mixed results. Two examples and an examination of Cochrane meta-analyses that used RR suggest bias in the results from the conventional inverse-variance-weighted approach. Finally, we comment on other measures of effect that have range restrictions, including risk difference, and outline further research.

Keywords: beta-binomial model; log-binomial model; relative risk; response ratio; risk difference.

Figure 1

Relation of estimates of the between‐studies variance (τ ²) to the overall log‐risk‐ratio (θ) in K studies, each of total sample size n, when data come from the binomial‐normal model with point mass for τ ² = 1 and π _jC = 0.1 (solid lines) and 0.3 (dashed). The Mandel‐Paule (circle), REML (triangle), and DerSimonian‐Laird (plus) estimators are compared with the true variance (cross). Light gray line at 1 [Colour figure can be viewed at wileyonlinelibrary.com]

Figure 2

Relation of estimates of the between‐studies variance τ ² to the overall log‐risk‐ratio (θ) in K studies, each of total sample size n, when data come from the binomial‐normal model with truncation for τ ² = 1 and π _jC = 0.1 (solid lines) and 0.3 (dashed). The Mandel‐Paule (circle), REML (triangle), and DerSimonian‐Laird (plus) estimation methods are compared with the true variance (cross). Light gray line at 1 [Colour figure can be viewed at wileyonlinelibrary.com]

Figure 3

Relation (to the overall log‐risk‐ratio, θ) of bias in the conventional method of estimating the log‐relative‐risk, θ, in the binomial‐normal model from K studies, each of total sample size n, with truncation (circle) or point‐mass (triangle) option, when τ ² (true value, τ ² = 1) is estimated by the Mandel‐Paule method, compared with true bias from truncation (cross) and point mass (diamond). π _jC = 0.1 (solid lines) and 0.3 (dashed). Light gray line at 0 [Colour figure can be viewed at wileyonlinelibrary.com]

Figure 4

Relation (to the overall log‐risk‐ratio, θ) of bias in estimating ρ from K studies, each of total size n, in the beta‐binomial model for ρ = 0.1 and π _C = 0.1 (solid lines) and 0.3 (dashed). The methods are Mandel‐Paule (circle), Breslow‐Day (cross), bbmle (reverse triangle), and gamlss (filled square). Light gray line at 0 [Colour figure can be viewed at wileyonlinelibrary.com]

Figure 5

Bias in estimating the overall log‐risk‐ratio, θ, from K studies, each of total size n, in the beta‐binomial model for ρ = 0.1 and π _C = 0.1 (solid lines) and 0.3 (dashed). The log‐relative‐risk is estimated by using inverse‐variance weights. The methods for estimation of ρ are Mandel‐Paule (circle), Breslow‐Day (cross), bbmle (reverse triangle), and gamlss (filled square). Light gray line at 0 [Colour figure can be viewed at wileyonlinelibrary.com]

Figure 6

Coverage of the overall log‐risk‐ratio, θ, from K studies, each of total size n, in the beta‐binomial model for ρ = 0.1 and π _C = 0.1 (solid lines) and 0.3 (dashed). The log‐relative‐risk is estimated by using inverse‐variance weights. The methods for estimation of ρ are Mandel‐Paule (circle), Breslow‐Day (cross), bbmle (reverse triangle), and gamlss (filled square). Light gray line at 0.95 [Colour figure can be viewed at wileyonlinelibrary.com]

Figure 7

Normal Q‐Q plot of the studentized residuals for the studies from random‐effects model (REM) meta‐analyses of log‐risk‐ratio with $\widehat{\theta }\ge 0$, ${\widehat{\tau }}^2>0$ in Cochrane Library Issue 4

Figure 8

Boxplots of studentized residuals by truncation probability, for the studies from random‐effects model (REM) meta‐analyses of log‐risk‐ratio with $\widehat{\theta }\ge 0$, ${\widehat{\tau }}^2>0$ in Cochrane Library Issue 4

Figure 9

Scatterplot (vs log‐risk‐ratio from fixed‐effect model [FEM]) of the meta‐analytic estimates of log‐risk‐ratio obtained by: (A) random‐effects model (REM), for the 353 REM meta‐analyses of risk ratio (RR) with ${\widehat{\theta }}_{\text{REM}}\ge 0$, ${\widehat{\tau }}^2>0$; (B) bbmle, for the 713 meta‐analyses of RR with ${\widehat{\theta }}_{bbmle}\ge 0$, $\widehat{\rho }>0$; (C) difference between log(RR) from REM and bbmle for the 353 meta‐analyses with $\widehat{\rho }>0$ and ${\widehat{\theta }}_{bbmle}\ge 0$ and τ ² > 0 and ${\widehat{\theta }}_{\text{REM}}\ge 0$