Living network meta-analysis compared with pairwise meta-analysis in comparative effectiveness research: empirical study

Abstract

Objective: To examine whether the continuous updating of networks of prospectively planned randomised controlled trials (RCTs) ("living" network meta-analysis) provides strong evidence against the null hypothesis in comparative effectiveness of medical interventions earlier than the updating of conventional, pairwise meta-analysis.

Design: Empirical study of the accumulating evidence about the comparative effectiveness of clinical interventions.

Data sources: Database of network meta-analyses of RCTs identified through searches of Medline, Embase, and the Cochrane Database of Systematic Reviews until 14 April 2015.

Eligibility criteria for study selection: Network meta-analyses published after January 2012 that compared at least five treatments and included at least 20 RCTs. Clinical experts were asked to identify in each network the treatment comparison of greatest clinical interest. Comparisons were excluded for which direct and indirect evidence disagreed, based on side, or node, splitting test (P<0.10).

Outcomes and analysis: Cumulative pairwise and network meta-analyses were performed for each selected comparison. Monitoring boundaries of statistical significance were constructed and the evidence against the null hypothesis was considered to be strong when the monitoring boundaries were crossed. A significance level was defined as α=5%, power of 90% (β=10%), and an anticipated treatment effect to detect equal to the final estimate from the network meta-analysis. The frequency and time to strong evidence was compared against the null hypothesis between pairwise and network meta-analyses.

Results: 49 comparisons of interest from 44 networks were included; most (n=39, 80%) were between active drugs, mainly from the specialties of cardiology, endocrinology, psychiatry, and rheumatology. 29 comparisons were informed by both direct and indirect evidence (59%), 13 by indirect evidence (27%), and 7 by direct evidence (14%). Both network and pairwise meta-analysis provided strong evidence against the null hypothesis for seven comparisons, but for an additional 10 comparisons only network meta-analysis provided strong evidence against the null hypothesis (P=0.002). The median time to strong evidence against the null hypothesis was 19 years with living network meta-analysis and 23 years with living pairwise meta-analysis (hazard ratio 2.78, 95% confidence interval 1.00 to 7.72, P=0.05). Studies directly comparing the treatments of interest continued to be published for eight comparisons after strong evidence had become evident in network meta-analysis.

Conclusions: In comparative effectiveness research, prospectively planned living network meta-analyses produced strong evidence against the null hypothesis more often and earlier than conventional, pairwise meta-analyses.

Fig 1

Efficacy of olanzapine versus haloperidol in treatment of acute schizophrenia, as estimated from living pairwise meta-analysis and living network meta-analysis. Monitoring boundaries were constructed using an α spending function with type I and type II errors fixed at 5% and 10%, respectively. Conventional significance thresholds are shown as dotted lines (z=1.96). The horizontal axis shows statistical information that accumulated over time, compared with maximum statistical information (information in single adequately powered study). Heterogeneity variance is assumed to be equal to the median of predictive distributions (0.049)

Fig 2

Kaplan-Meier survival curves for non-strong evidence against null hypothesis, comparing sequential pairwise and network meta-analysis of 49 comparisons. Events occur when monitoring boundaries are crossed for comparison of interest. Time is measured as years from time point when both interventions are included in network