Much ado about nothing: a comparison of the performance of meta-analytical methods with rare events

Abstract

For rare outcomes, meta-analysis of randomized trials may be the only way to obtain reliable evidence of the effects of healthcare interventions. However, many methods of meta-analysis are based on large sample approximations, and may be unsuitable when events are rare. Through simulation, we evaluated the performance of 12 methods for pooling rare events, considering estimability, bias, coverage and statistical power. Simulations were based on data sets from three case studies with between five and 19 trials, using baseline event rates between 0.1 and 10 per cent and risk ratios of 1, 0.75, 0.5 and 0.2. We found that most of the commonly used meta-analytical methods were biased when data were sparse. The bias was greatest in inverse variance and DerSimonian and Laird odds ratio and risk difference methods, and the Mantel-Haenszel (MH) odds ratio method using a 0.5 zero-cell correction. Risk difference meta-analytical methods tended to show conservative confidence interval coverage and low statistical power at low event rates. At event rates below 1 per cent the Peto one-step odds ratio method was the least biased and most powerful method, and provided the best confidence interval coverage, provided there was no substantial imbalance between treatment and control group sizes within trials, and treatment effects were not exceptionally large. In other circumstances the MH OR without zero-cell corrections, logistic regression and the exact method performed similarly to each other, and were less biased than the Peto method.