Common pitfalls and mistakes in the set-up, analysis and interpretation of results in network meta-analysis: what clinicians should look for in a published article

Abstract

Objective: Several tools have been developed to evaluate the extent to which the findings from a network meta-analysis would be valid; however, applying these tools is a time-consuming task and often requires specific expertise. Clinicians have little time for critical appraisal, and they need to understand the key elements that help them select network meta-analyses that deserve further attention, optimising time and resources. This paper is aimed at providing a practical framework to assess the methodological robustness and reliability of results from network meta-analysis.

Methods: As a working example, we selected a network meta-analysis about drug treatments for generalised anxiety disorder, which was published in 2011 in the British Medical Journal. The same network meta-analysis was previously used to illustrate the potential of this methodology in a methodological paper published in JAMA.

Results: We reanalysed the 27 studies included in this network following the methods reported in the original article and compared our findings with the published results. We showed how different methodological approaches and the presentation of results can affect conclusions from network meta-analysis. We divided our results into three sections, according to the specific issues that should always be addressed in network meta-analysis: (1) understanding the evidence base, (2) checking the statistical analysis and (3) checking the reporting of findings.

Conclusions: The validity of the results from network meta-analysis depends on the plausibility of the transitivity assumption. The risk of bias introduced by limitations of individual studies must be considered first and judgement should be used to infer about the plausibility of transitivity. Inconsistency exists when treatment effects from direct and indirect evidence are in disagreement. Unlike transitivity, inconsistency can be always evaluated statistically, and it should be specifically investigated and reported in the published paper. Network meta-analysis allows researchers to list treatments in preferential order; however, in this paper we demonstrated that rankings could be misleading if based on the probability of being the best. Clinicians should always be interested in the effect sizes rather than the naive rankings.

Keywords: Anxiety Disorders; Clinical Trials; Mental Health.

Figure 1

Percentage contribution of the Silverstone study in the network estimates of the relative effects for all interventions versus fluoxetine for the response outcome. The percentage contribution of the included studies was estimated assuming for all comparisons the heterogeneity standard deviation estimate obtained from the Bayesian hierarchical model (=0.26).

Figure 2

Inconsistency results of the loop-specific approach for the outcome of response. Squares represent the ratio of ORs (RORs) between the direct and indirect estimates and the black horizontal lines the respective 95% CI truncated to the null value of 1 (red line). DULO, duloxetine; ESC, escitalopram; FLUO, fluoxetine; LOR, larazepam; PAR, paroxetine; PLA, placebo; PREG, pregabaline; SER, sertraline; VEN, venlafaxine.

Figure 3

Median ORs of all active treatments versus placebo for the response outcome. The black horizontal lines represent the 95% credible intervals (CrI) and the red lines represent the respective 95% predictive intervals (PrI). The vertical blue dotted line shows the null value (OR=1).