Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Nov 23;19(1):304.
doi: 10.1186/s12916-021-02166-3.

ROB-MEN: a tool to assess risk of bias due to missing evidence in network meta-analysis

Virginia Chiocchia  1   2 , Adriani Nikolakopoulou  3   4 , Julian P T Higgins  5 , Matthew J Page  6 , Theodoros Papakonstantinou  3   4 , Andrea Cipriani  7   8 , Toshi A Furukawa  9 , George C M Siontis  10 , Matthias Egger  3   5   11 , Georgia Salanti  3
Affiliations

ROB-MEN: a tool to assess risk of bias due to missing evidence in network meta-analysis

Virginia Chiocchia et al. BMC Med. .

Abstract

Background: Selective outcome reporting and publication bias threaten the validity of systematic reviews and meta-analyses and can affect clinical decision-making. A rigorous method to evaluate the impact of this bias on the results of network meta-analyses of interventions is lacking. We present a tool to assess the Risk Of Bias due to Missing Evidence in Network meta-analysis (ROB-MEN).

Methods: ROB-MEN first evaluates the risk of bias due to missing evidence for each of the possible pairwise comparison that can be made between the interventions in the network. This step considers possible bias due to the presence of studies with unavailable results (within-study assessment of bias) and the potential for unpublished studies (across-study assessment of bias). The second step combines the judgements about the risk of bias due to missing evidence in pairwise comparisons with (i) the contribution of direct comparisons to the network meta-analysis estimates, (ii) possible small-study effects evaluated by network meta-regression, and (iii) any bias from unobserved comparisons. Then, a level of "low risk", "some concerns", or "high risk" for the bias due to missing evidence is assigned to each estimate, which is our tool's final output.

Results: We describe the methodology of ROB-MEN step-by-step using an illustrative example from a published NMA of non-diagnostic modalities for the detection of coronary artery disease in patients with low risk acute coronary syndrome. We also report a full application of the tool on a larger and more complex published network of 18 drugs from head-to-head studies for the acute treatment of adults with major depressive disorder.

Conclusions: ROB-MEN is the first tool for evaluating the risk of bias due to missing evidence in network meta-analysis and applies to networks of all sizes and geometry. The use of ROB-MEN is facilitated by an R Shiny web application that produces the Pairwise Comparisons and ROB-MEN Table and is incorporated in the reporting bias domain of the CINeMA framework and software.

Keywords: Evidence synthesis; Missing evidence; Network meta-analysis; Publication bias; Reporting bias; Risk of bias; Selective outcome reporting.

PubMed Disclaimer

Conflict of interest statement

All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: AC has received research and consultancy fees from INCiPiT (Italian Network for Paediatric Trials), CARIPLO Foundation and Angelini Pharma; TAF reports personal fees from MSD, grants and personal fees from Mitsubishi-Tanabe, grants and personal fees from Shionogi, outside the submitted work; TAF has a patent 2018-177688 pending, and a patent Kokoro-app issued; no other relationships or activities that could appear to have influenced the submitted work.

Figures

Fig. 1
Fig. 1
Network plots of network meta-analysis of non-invasive diagnostic modalities for detecting coronary artery disease. a Standard network plot. b Network graph showing risk of bias assessment for pairwise comparisons. Sizes of solid lines and nodes are proportional to number of studies in each comparison and total sample size for each treatment, respectively. Solid lines represent the observed direct comparisons, dotted lines represent unobserved comparisons between interventions. Green indicates no bias detected, orange indicates suspected bias favouring the treatment indicated by the arrow. ECG: electrocardiogram; CCTA: coronary computed tomographic angiography; CMR: cardiovascular magnetic resonance; SPECT-MPI: single-photon emission computed tomography-myocardial perfusion imaging; Stress Echo: stress echocardiography
Fig. 2
Fig. 2
Overview of the ROB-MEN process
See this image and copyright information in PMC

References

    1. Page MJ, Higgins JP, Sterne JA. Cochrane Handbook for Systematic Reviews of Interventions. version 6.0. Cochrane. 2019. Chapter 13: Assessing risk of bias due to missing results in a synthesis.
    1. Egger M, Smith GD, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315(7109):629–634. doi: 10.1136/bmj.315.7109.629. - DOI - PMC - PubMed
    1. Sterne JAC, Egger M. Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. J Clin Epidemiol. 2001;54(10):1046–1055. doi: 10.1016/S0895-4356(01)00377-8. - DOI - PubMed
    1. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. J Clin Epidemiol. 2008;61(10):991–996. doi: 10.1016/j.jclinepi.2007.11.010. - DOI - PubMed
    1. Sterne JAC, Sutton AJ, Ioannidis JPA, Terrin N, Jones DR, Lau J, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343 jul22 1:d4002. doi: 10.1136/bmj.d4002. - DOI - PubMed

Publication types