Evaluating the quality of evidence from a network meta-analysis

Abstract

Systematic reviews that collate data about the relative effects of multiple interventions via network meta-analysis are highly informative for decision-making purposes. A network meta-analysis provides two types of findings for a specific outcome: the relative treatment effect for all pairwise comparisons, and a ranking of the treatments. It is important to consider the confidence with which these two types of results can enable clinicians, policy makers and patients to make informed decisions. We propose an approach to determining confidence in the output of a network meta-analysis. Our proposed approach is based on methodology developed by the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group for pairwise meta-analyses. The suggested framework for evaluating a network meta-analysis acknowledges (i) the key role of indirect comparisons (ii) the contributions of each piece of direct evidence to the network meta-analysis estimates of effect size; (iii) the importance of the transitivity assumption to the validity of network meta-analysis; and (iv) the possibility of disagreement between direct evidence and indirect evidence. We apply our proposed strategy to a systematic review comparing topical antibiotics without steroids for chronically discharging ears with underlying eardrum perforations. The proposed framework can be used to determine confidence in the results from a network meta-analysis. Judgements about evidence from a network meta-analysis can be different from those made about evidence from pairwise meta-analyses.

Figure 1. Network of topical antibiotics without steroids for chronically discharging ears.

Edges are weighted according to the inverse of the variance of the direct summary ln(OR) (presented along the edges) and nodes are weighted according to the number of studies.

Figure 2. Contributions matrix: percentage contribution of each direct estimate to the network meta-analysis estimates.

Rows correspond to network meta-analysis ORs (separated for mixed and indirect evidence) and columns correspond to direct meta-analysis ORs. The contribution of each direct comparison to the total network evidence that provides the ranking of the treatments is presented separately (row named Entire network). The sizes of the boxes are proportional to the percentage contribution of each direct estimate to the network meta-analysis estimates (rows 1–6) and to the entire network (row 7). The last row shows the number of included direct comparisons. The names of the treatments are given in Figure 1.

Figure 3. Study limitations for each network estimate for pairwise comparisons of topical antibiotics.

Calculations are based on the contributions of direct evidence. The colours represent the risk of bias (green: low, yellow: moderate, red: high). The initial judgements about the risk of bias in the direct estimates are shown on the right side of the figure (there is no direct evidence for AC). The names of the treatments are given in Figure 1.

Figure 4. Network estimates of mean ORs, their 95% confidence intervals and 95% predictive intervals (red extensions).

The names of the treatments are given in Figure 1.

Figure 5. Study limitations weighted by contribution of direct estimates to the network of topical antibiotics.

The colours represent the risk of bias (green: low, yellow: moderate, red: high). The names of the treatments are given in Figure 1.

Figure 6. Rankograms for topical antibiotics without steroids for chronically discharging ears.

On the horizontal axes are the possible ranks and on the vertical axis the probability that each treatment achieves each rank.

Figure 7. Comparison-adjusted funnel plot for the network of topical antibiotics without steroids for chronically discharging ears.

Each observation is the difference between a study estimate and its direct meta-analysis mean effect. Studies on the right hand side ‘overestimate’ the effect of newer treatments.