Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Feb;28(1):58-67.
doi: 10.1136/bmjebm-2022-111944. Epub 2022 Aug 10.

Knowledge user survey and Delphi process to inform development of a new risk of bias tool to assess systematic reviews with network meta-analysis (RoB NMA tool)

Carole Lunny  1   2 , Areti Angeliki Veroniki  3   4 , Brian Hutton  5   6 , Ian White  7 , Jpt Higgins  8 , James M Wright  2 , Ji Yoon Kim  9 , Sai Surabi Thirugnanasampanthar  10 , Shazia Siddiqui  3 , Jennifer Watt  11 , Lorenzo Moja  12 , Nichole Taske  13 , Robert C Lorenz  14 , Savannah Gerrish  15 , Sharon Straus  3   11 , Virginia Minogue  16 , Franklin Hu  17 , Kevin Lin  17 , Ayah Kapani  17 , Samin Nagi  17 , Lillian Chen  17 , Mona Akbar-Nejad  17 , Andrea C Tricco  3   4
Affiliations

Knowledge user survey and Delphi process to inform development of a new risk of bias tool to assess systematic reviews with network meta-analysis (RoB NMA tool)

Carole Lunny et al. BMJ Evid Based Med. 2023 Feb.

Abstract

Background: Network meta-analysis (NMA) is increasingly used in guideline development and other aspects of evidence-based decision-making. We aimed to develop a risk of bias (RoB) tool to assess NMAs (RoB NMA tool). An international steering committee recommended that the RoB NMA tool to be used in combination with the Risk of Bias in Systematic reviews (ROBIS) tool (i.e. because it was designed to assess biases only) or other similar quality appraisal tools (eg, A MeaSurement Tool to Assess systematic Reviews 2 [AMSTAR 2]) to assess quality of systematic reviews. The RoB NMA tool will assess NMA biases and limitations regarding how the analysis was planned, data were analysed and results were presented, including the way in which the evidence was assembled and interpreted.

Objectives: Conduct (a) a Delphi process to determine expert opinion on an item's inclusion and (b) a knowledge user survey to widen its impact.

Design: Cross-sectional survey and Delphi process.

Methods: Delphi panellists were asked to rate whether items should be included. All agreed-upon item were included in a second round of the survey (defined as 70% agreement). We surveyed knowledge users' views and preferences about the importance, utility and willingness to use the RoB NMA tool to evaluate evidence in practice and in policymaking. We included 12 closed and 10 open-ended questions, and we followed a knowledge translation plan to disseminate the survey through social media and professional networks.

Results: 22 items were entered into a Delphi survey of which 28 respondents completed round 1, and 22 completed round 2. Seven items did not reach consensus in round 2. A total of 298 knowledge users participated in the survey (14% respondent rate). 75% indicated that their organisation produced NMAs, and 78% showed high interest in the tool, especially if they had received adequate training (84%). Most knowledge users and Delphi panellists preferred a tool to assess both bias in individual NMA results and authors' conclusions. Response bias in our sample is a major limitation as knowledge users working in high-income countries were more represented. One of the limitations of the Delphi process is that it depends on the purposive selection of experts and their availability, thus limiting the variability in perspectives and scientific disciplines.

Conclusions: This Delphi process and knowledge user survey informs the development of the RoB NMA tool.

Keywords: evidence-based practice; health care quality, access, and evaluation; health services research; methods.

PubMed Disclaimer

Conflict of interest statement

Competing interests: AAV was an Associate Editor for the journal, but was not involved with the decision or peer-review process.

Figures

Figure 1
Figure 1
(A) Use of a risk of bias (RoB) network meta-analysis (NMA) tool in knowledge users work and (B) interest in a RoB NMA tool. The left figure (A) depicts the proportion of responses to the question of whether knowledge users would use our proposed RoB NMA tool to assess the NMA analysis results, the authors’ conclusions or both results and conclusions. The right figure (B) shows the proportion of responses to the question about interest in a tool for appraising RoB in NMAs.
Figure 2
Figure 2
Flow chart of emails sent and responses.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Moher D, Schulz KF, Simera I, et al. . Guidance for developers of health research reporting guidelines. PLoS Med 2010;7:e1000217. 10.1371/journal.pmed.1000217 - DOI - PMC - PubMed
    1. Whiting P, Wolff R, Mallett S, et al. . A proposed framework for developing quality assessment tools. Syst Rev 2017;6:204. 10.1186/s13643-017-0604-6 - DOI - PMC - PubMed
    1. Bullock A, Morris ZS, Atwell C. Collaboration between health services managers and researchers: making a difference? J Health Serv Res Policy 2012;17 Suppl 2:2–10. 10.1258/jhsrp.2011.011099 - DOI - PubMed
    1. Hanney SR, Gonzalez-Block MA, Buxton MJ, et al. . The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst 2003;1:1–28. 10.1186/1478-4505-1-2 - DOI - PMC - PubMed
    1. Innvaer S, Vist G, Trommald M, et al. . Health policy-makers' perceptions of their use of evidence: a systematic review. J Health Serv Res Policy 2002;7:239–44. 10.1258/135581902320432778 - DOI - PubMed