Sensitivity of treatment recommendations to bias in network meta-analysis

Abstract

Network meta-analysis (NMA) pools evidence on multiple treatments to estimate relative treatment effects. Included studies are typically assessed for risk of bias; however, this provides no indication of the impact of potential bias on a decision based on the NMA. We propose methods to derive bias adjustment thresholds which measure the smallest changes to the data that result in a change of treatment decision. The methods use efficient matrix operations and can be applied to explore the consequences of bias in individual studies or aggregate treatment contrasts, in both fixed and random-effects NMA models. Complex models with multiple types of data input are handled by using an approximation to the hypothetical aggregate likelihood. The methods are illustrated with a simple NMA of thrombolytic treatments and a more complex example comparing social anxiety interventions. An accompanying R package is provided.

Keywords: Evidence synthesis; Influence matrix; Mixed treatment comparison; Quality of evidence; Risk of bias; Threshold analysis.

Figure 1

Example construction of a decision invariant bias adjustment interval () for a data point y _m, for an NMA with five treatments and current optimal treatment $k^{*}\text{=}4$: the new treatment decision at the negative and positive thresholds would be 2 and 3 respectively

Figure 2

Example of thresholds lines in two dimensions for an NMA of five treatments with k ^*=4 (, invariant region about the origin (no bias adjustment)): any simultaneous bias adjustment $({\mathit{\beta }}_{m_1},{\mathit{\beta }}_{m_2})$ to data points $y_{m_1}$ and $y_{m_2}$ which remains within the invariant region does not change the optimal treatment; at the boundary of the invariant region formed by thresholds line ${\mathit{\beta }}_{a{k}^{*}}^{\mathrm{thresh}}$ the new optimal treatment is $\tilde{k}{}^{*}=a$

Figure 3

Thrombolytics example network, showing how the six treatments are connected by study evidence: nodes represent treatments and edges show comparisons made by studies; numbers inside the nodes are the treatment codings; numbers on the edges give the number of studies making that comparison; the bold triangle is the loop formed by the three‐arm study

Figure 4

Study level forest plot, displaying invariant intervals for the thrombolytics example, sorted with smallest thresholds first (bold labels in the table emphasize study estimates with short invariant intervals lying within the 95% CI; the optimal treatment without bias adjustment is k*=3): ∘, log‐OR;—, 95% CI; , invariant interval

Figure 5

Contrast level forest plot, displaying invariant intervals for the thrombolytics example (bold labels in the table emphasize contrast estimates with short invariant intervals lying within the 95% credible interval; the optimal treatment without bias adjustment is k*=3): ∘, log‐OR;—, 95% credible interval; , invariant interval

Figure 6

Invariant region formed from threshold lines for bias adjustment to the two relative effect estimates from study 1 (, invariant region): the new optimal treatments on the boundary are indicated by ${\tilde{k}}^{*}$; optimal treatment without bias adjustment is k ^*=3

Figure 7

Social anxiety treatment network: nodes represent treatments and edges show study comparisons; numbers around the edge are the treatment codings; treatment classes are indicated by the braces (some classes contain a single treatment only); treatment 1 is waitlist, treatment 2 is pill placebo and treatment 3 is psychological placebo; Table A1 in the on‐line appendix A.12 lists the treatment codes and classes

Figure 8

Contrast level forest plot for the social anxiety example showing results of the threshold analysis, sorted with smallest thresholds first (only contrasts with a threshold less than 2 SMDs are shown here for brevity; the complete results can be found in the Web supplementary material; the optimal treatment without bias adjustment is k*=41); NT, no threshold; ∘, SMD; —, 95% credible interval; , invariant interval

Figure 9

(a) Invariant interval for all pharmacological treatments against an inactive control, considered to be bias adjusted by the same amount, and (b) invariant interval for all psychological treatments against an inactive control, considered to be bias adjusted by the same amount; the optimal treatment without any bias adjustment is k*=41

Figure 10

Invariant region () for simultaneous adjustments for common biases in all psychological and all pharmacological treatments: the new treatment recommendation at the boundary is shown as ${\tilde{k}}^{*}$; the optimal treatment without bias adjustment is k ^*=41