Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2025 Mar 18:388:e079839.
doi: 10.1136/bmj-2024-079839.

Risk of Bias in Network Meta-Analysis (RoB NMA) tool

Carole Lunny  1   2   3 , J P T Higgins  4   5 , Ian R White  6 , Sofia Dias  7 , B Hutton  8   9 , J M Wright  3 , Areti-Angeliki Veroniki  2   10 , P F Whiting  11 , A C Tricco  2   12   13
Affiliations

Risk of Bias in Network Meta-Analysis (RoB NMA) tool

Carole Lunny et al. BMJ. .

Erratum in

Abstract

Systematic reviews with network meta-analysis (NMA) have potential biases in their conduct, analysis, and interpretation. If the results or conclusions of an NMA are integrated into policy or practice without any consideration of risks of bias, decisions could unknowingly be based on incorrect results, which could translate to poor patient outcomes. The RoB NMA (Risk of Bias in Network Meta-Analysis) tool answers a clearly defined need for a rigorously developed tool to assess risk of bias in NMAs of healthcare interventions. In this guidance article, we describe and provide a justification for the tool’s 17 items, their mechanism of bias, pertinent examples, and how to assess an NMA based on each response option.

PubMed Disclaimer

Conflict of interest statement

Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/disclosure-of-interest/ and declare: support from a 2020 CIHR Project Grant, Medical Research Council, Eversana, and National Institute for Health Research (NIHR) Applied Research Collaboration West (ARC West) for the submitted work; no support from any organisation for the submitted work; no financial relationships with any organisations that may have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

Figures

Fig 1
Fig 1
Connected network (A-C) and disconnected network (D). No trials connect intervention E or F to the rest of the interventions, making diagram D disconnected
Fig 2
Fig 2
Processes and tools used by authors of systematic reviews with network meta-analysis (NMA), or assessors of completed or published systematic reviews with network meta-analysis. Left: tools that authors use when conducting a systematic review with NMA (eg, Cochrane RoB 2), and when completed and published, assessors can use the tools (right) to assess the systematic review with NMA for biases (eg, ROBIS). Authors of NMAs should first conduct the systematic review using established guidance documents. When all studies have been included, the Cochrane RoB 2 tool can be used to assess the risk of bias of randomised controlled trials (or ROBINS-I for observational studies). RoB-ME and ROB-MEN tools can also be used to evaluate the systematic review with or without NMA. These assessments by NMA authors can then feed into a CINeMA or GRADE NMA assessment of the certainty of the evidence, which includes evaluation of risk of bias, imprecision, indirectness, heterogeneity, and inconsistency. Reporting checklists, such as PRISMA-NMA, help NMA authors when they have completed their review and are ready to publish. The reporting checklists help describe the review and NMA comprehensively, transparently, and accurately. Once the publication or report is complete and publicly available, individuals (ie, assessors) can use the ROBIS and RoB NMA tools to assess the systematic review with NMA for known meta level biases, such as publication bias (eg, where studies are missing from the published literature) and selective non-reporting of outcomes or analyses (eg, because they did not reach a desired level of magnitude or statistical significance). *Established methodological guidance for conducting systematic reviews includes the Cochrane Handbook for Systematic Reviews of Interventions, JBI Manual for Evidence Synthesis, or Cochrane-Campbell Handbook for Qualitative Evidence Synthesis. These guidance documents have detailed steps on how to plan, conduct, organise, and present a systematic review. †Or a suitable alternative for non-randomised studies (ie, ROBINS-I). ‡Or other suitable tools (ie, AMSTAR 2 (A Measurement Tool to Assess Systematic Reviews, version 2)). CINeMA=Confidence in Network Meta-Analysis; GRADE NMA=Grading of Recommendations Assessment, Development, and Evaluation in a Network Meta-Analysis; PRISMA=Preferred Reporting Items for Systematic reviews and Meta-Analyses; ROB ME=Risk of Bias due to Missing Evidence; ROB-MEN=Risk of Bias due to Missing Evidence in Network Meta-Analysis; RoB NMA=Risk of Bias in Network Meta-Analysis; RoB 2=revised Cochrane Risk of Bias tool for randomised trials; ROBINS-I=Risk of Bias in Non-randomised Studies of Interventions
Fig 3
Fig 3
Flow diagram of RoB NMA (Risk of Bias in Network Meta-Analysis) tool development process
Fig 4
Fig 4
Process for using the ROBIS and then the RoB NMA (Risk of Bias in Network Meta-Analysis) tools to reach overall judgments on two different network meta-analyses (NMAs) in the same systematic review (SR). With the judgments in the first three domains of ROBIS and the three domains of RoB NMA, the assessor then makes an overall judgment about the potential for bias in one or multiple NMA results (eg, one outcome from one network) within an SR. Assessors can also make an overall judgment of the bias in the authors’ conclusions by assessing the interpretation of the findings
Fig 5
Fig 5
Example format for an overall ROBIS (a tool for assessing risk of bias in systematic reviews) and RoB NMA (Risk of Bias in Network Meta-Analysis) assessment. ID=identification; SR=systematic review
See this image and copyright information in PMC

References

    1. Chaimani A, Caldwell DM, Li T, Higgins JP, Salanti G. Undertaking network meta‐analyses. Cochrane Database Syst Rev 2019;285-320.
    1. Chaimani A, Salanti G, Leucht S, Geddes JR, Cipriani A. Common pitfalls and mistakes in the set-up, analysis and interpretation of results in network meta-analysis: what clinicians should look for in a published article. Evid Based Ment Health 2017;20:88-94. 10.1136/eb-2017-102753 - DOI - PMC - PubMed
    1. Salanti G. Indirect and mixed-treatment comparison, network, or multiple-treatments meta-analysis: many names, many benefits, many concerns for the next generation evidence synthesis tool. Res Synth Methods 2012;3:80-97. 10.1002/jrsm.1037 - DOI - PubMed
    1. Salanti G, Del Giovane C, Chaimani A, Caldwell DM, Higgins JP. Evaluating the quality of evidence from a network meta-analysis. PLoS One 2014;9:e99682. 10.1371/journal.pone.0099682 - DOI - PMC - PubMed
    1. Salanti G, Ades AE, Ioannidis JP. Graphical methods and numerical summaries for presenting results from multiple-treatment meta-analysis: an overview and tutorial. J Clin Epidemiol 2011;64:163-71. 10.1016/j.jclinepi.2010.03.016 - DOI - PubMed

LinkOut - more resources