Simulation evaluation of statistical properties of methods for indirect and mixed treatment comparisons

Abstract

Background: Indirect treatment comparison (ITC) and mixed treatment comparisons (MTC) have been increasingly used in network meta-analyses. This simulation study comprehensively investigated statistical properties and performances of commonly used ITC and MTC methods, including simple ITC (the Bucher method), frequentist and Bayesian MTC methods.

Methods: A simple network of three sets of two-arm trials with a closed loop was simulated. Different simulation scenarios were based on different number of trials, assumed treatment effects, extent of heterogeneity, bias and inconsistency. The performance of the ITC and MTC methods was measured by the type I error, statistical power, observed bias and mean squared error (MSE).

Results: When there are no biases in primary studies, all ITC and MTC methods investigated are on average unbiased. Depending on the extent and direction of biases in different sets of studies, ITC and MTC methods may be more or less biased than direct treatment comparisons (DTC). Of the methods investigated, the simple ITC method has the largest mean squared error (MSE). The DTC is superior to the ITC in terms of statistical power and MSE. Under the simulated circumstances in which there are no systematic biases and inconsistencies, the performances of MTC methods are generally better than the performance of the corresponding DTC methods. For inconsistency detection in network meta-analysis, the methods evaluated are on average unbiased. The statistical power of commonly used methods for detecting inconsistency is very low.

Conclusions: The available methods for indirect and mixed treatment comparisons have different advantages and limitations, depending on whether data analysed satisfies underlying assumptions. To choose the most valid statistical methods for research synthesis, an appropriate assessment of primary studies included in evidence network is required.

Figure 1

Network of simulated trials.

Figure 2

Mean squared error (MSE) by different comparison models (Note: baseline risk 20%; zero treatment effect; without systematic bias in trials; fix, fixed effect; ran, random-effects; Tau2 refers τ²).

Figure 3

Bias by different comparison methods (Note: selected simulation scenarios, baseline risk = 20%; τ² = 0.05; number of studies =3x20; random-effects analyses).

Figure 4

Type I error – proportion of significant results when true treatment effect is zero, impact of number of studies and assumed heterogeneity (Note: baseline risk =20%; fix, fixed effect; ran, random-effects; Tau2 refers τ²).

Figure 5

Statistical power to detect treatment effect (OR23 = 0.75), impact of number of studies and assumed heterogeneity (Note: Baseline risk =20%; fix, fixed effect; ran, random-effects; Tau2 refers τ²).

Figure 6

Estimated inconsistency (log ROR) – a selected simulation scenario (Note: true logROR = 0.223; baseline risk = 20%; number of studies =3x20; τ² = 0.10; Tau2 refers τ²).

Figure 7

Type I error for inconsistency detection: impact of heterogeneity and number of studies (Note: baseline risk =20%, true lnROR = 0; tau2 refers τ²; Freq-fix, frequentist fixed-effect; Freq-ran, frequentist random-effects; Bay-fix, Bayesian fixed-effect; Bay-ran, Bayesian random-effects).

Figure 8

Statistical power to detect inconsistency: impact of heterogeneity and number of studies (Note: baseline risk =20%, true lnROR = 0.223; tau2 refers τ². Freq-fix, frequentist fixed-effect; Freq-ran, frequentist random-effects; Bay-fix, Bayesian fixed-effect; Bay-ran, Bayesian random-effects).