Real-world Performance of Meta-analysis Methods for Double-Zero-Event Studies with Dichotomous Outcomes Using the Cochrane Database of Systematic Reviews

Abstract

Background: Meta-analysis combines multiple independent studies, which can increase power and provide better estimates. However, it is unclear how best to deal with studies with zero events; such studies are also known as double-zero-event studies (DZS). Several statistical methods have been proposed, but the agreement among different approaches has not been systematically assessed using real-world published systematic reviews.

Methods: The agreement of five commonly used methods (i.e., the inverse-variance, Mantel-Haenszel, Peto, Bayesian, and exact methods) was assessed using the Cohen's κ coefficients using 368 meta-analyses with rare events selected from the Cochrane Database of Systematic Reviews. Three continuity corrections, including the correction of a constant 0.5, the treatment arm continuity correction (TACC), and the empirical (EMP) correction, were used to handle DZS when applying inverse-variance and Mantel-Haenszel methods.

Results: When the proportion of DZS studies was lower than 50% in a meta-analysis, different methods had moderately high agreement. However, when this proportion was increased to be over 50%, the agreement among the methods decreased to different extents. For the Bayesian, exact, and Peto methods and the inverse-variance and Mantel-Haenszel methods using the EMP correction, their agreement coefficients with the inverse-variance and Mantel-Haenszel methods using a constant 0.5 and TACC decreased from larger than 0.70 to smaller than 0.30. In contrast, the agreement coefficients only decreased slightly among the Bayesian, exact, and Peto methods and the inverse-variance and Mantel-Haenszel methods using the EMP correction.

Conclusions: To utilize all available information and reduce research waste and avoid overestimating the effect, meta-analysts should incorporate DZS, rather than simply removing them. The Peto and other conventional methods with continuity correction should be avoided when the proportion of DZS is extremely high. The exact and Bayesian methods are highly recommended, except when none of the included studies have an event in one or both treatment arms.

Keywords: Cochrane Database of Systematic Reviews; Cohen’s κ coefficient; double-zero-event studies; meta-analysis; rare event.

Figure 1

Flow chart of the selection criteria for the eligible meta-analyses from the CDSR. AE, adverse events; CDSR, Cochrane Database of Systematic Reviews; CONT, continuous; DICH, dichotomous; MA, meta-analysis.

Figure 2

Scatter plot matrix of the agreement between the estimated overall log odds ratios produced by the fixed effect methods that include double-zero-event studies. I5, inverse-variance method with 0.5 continuity correction; ITACC, inverse-variance method with treatment arm continuity correction; IEMP, inverse-variance method with empirical continuity correction; MH5, Mantel–Haenszel method with 0.5 continuity correction; MHTACC, Mantel–Haenszel method with treatment arm continuity correction; MHEMP, Mantel–Haenszel method with empirical continuity correction.

Figure 3

Scatter plot matrix of the agreement between the lengths of 95% confidence intervals produced by the fixed effect methods that include double-zero-event studies. I5, inverse-variance method with 0.5 continuity correction; ITACC, inverse-variance method with treatment arm continuity correction; IEMP, inverse-variance method with empirical continuity correction; MH5, Mantel–Haenszel method with 0.5 continuity correction; MHTACC, Mantel–Haenszel method with treatment arm continuity correction; MHEMP, Mantel–Haenszel method with empirical continuity correction.