Evaluation of inconsistency in networks of interventions

Abstract

Background: The assumption of consistency, defined as agreement between direct and indirect sources of evidence, underlies the increasingly popular method of network meta-analysis. No evidence exists so far regarding the extent of inconsistency in full networks of interventions or the factors that control its statistical detection.

Methods: In this paper we assess the prevalence of inconsistency from data of 40 published networks of interventions involving 303 loops of evidence. Inconsistency is evaluated in each loop by contrasting direct and indirect estimates and by employing an omnibus test of consistency for the entire network. We explore whether different effect measures for dichotomous outcomes are associated with differences in inconsistency, and evaluate whether different ways to estimate heterogeneity affect the magnitude and detection of inconsistency.

Results: Inconsistency was detected in from 2% to 9% of the tested loops, depending on the effect measure and heterogeneity estimation method. Loops that included comparisons informed by a single study were more likely to show inconsistency. About one-eighth of the networks were found to be inconsistent. The proportions of inconsistent loops do not materially change when different effect measures are used. Important heterogeneity or the overestimation of heterogeneity was associated with a small decrease in the prevalence of statistical inconsistency.

Conclusions: The study suggests that changing the effect measure might improve statistical consistency, and that an analysis of sensitivity to the assumptions and an estimator of heterogeneity might be needed before reaching a conclusion about the absence of statistical inconsistency, particularly in networks with few studies.

Appendix Figure 1

The left-hand side panels represent a plot of inconsistency estimate ($\widehat{\mathit{IF}}$) versus the heterogeneity variance (${\widehat{\tau }}^2$) and the right-hand side panels correspond to a plot of the P value of IF versus ${\widehat{\tau }}^2$. Inconsistency is estimated under the common within-loop heterogeneity variance and under the DerSimonian and Laird (DL), restricted maximum likelihood (REML) and Sidik-Jonkman (SJ) methods.

Figure 1

Flow chart of the process of selecting articles describing network analyses.

Figure 2

Plot of the two sided P values of IF (fourth-root scale) for OR vs. RD, OR vs. RRH and OR vs. RRB effect measures under the DerSimonian and Laird method for ${\tau }_{\mathit{loop}}^2$ and the restricted maximum likelihood for ${\tau }_{\mathit{ntw}}^2$. The solid diagonal line indicates equality, the dashed diagonal line is the regression line and the two dotted horizontal and vertical lines represent the P=0.05 threshold lines. RD is the risk difference measure, RRH is the risk ratio for harmful outcomes, RRB is the risk ratio for beneficial outcomes and OR is the odds ratio.

Figure 3

Comparison of the estimated heterogeneity variance under the DerSimonian and Laird (DL), restricted maximum likelihood (REML) and Sidik-Jonkman (SJ) methods on the log scale when applying the loop-specific approach (common within-loop heterogeneity variance, ${\tau }_{\mathit{loop}}^2$) in the 303 loops.

Figure 4

Histogram of the absolute values of the inconsistency factors (IF) for the OR effect measure estimated under the common within-loop heterogeneity variance, ${\tau }_{\mathit{loop}}^2$, estimated with the DerSimonian and Laird method.

Figure 5

Plot of heterogeneity estimates from the consistency model against heterogeneity estimates from the inconsistency model under the design-by-treatment interaction approach, along with the equality line. Heterogeneity is estimated under maximum likelihood (1st panel) and restricted maximum likelihood (2nd panel) methods when the effect measure is the odds ratio (OR).