Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Meta-Analysis
. 2011 Feb 24;6(2):e17242.
doi: 10.1371/journal.pone.0017242.

Applying the risk of bias tool in a systematic review of combination long-acting beta-agonists and inhaled corticosteroids for persistent asthma

Lisa Hartling  1 , Kenneth Bond, Ben Vandermeer, Jennifer Seida, Donna M Dryden, Brian H Rowe
Affiliations
Meta-Analysis

Applying the risk of bias tool in a systematic review of combination long-acting beta-agonists and inhaled corticosteroids for persistent asthma

Lisa Hartling et al. PLoS One. .

Abstract

Background: The Risk of Bias (RoB) tool is used to assess internal validity of randomized controlled trials (RCTs). Our objectives were to: 1) evaluate inter-rater agreement of the RoB tool; 2) determine the time to access supplemental study information; 3) compare the RoB tool with the Jadad scale and Schulz allocation concealment (AC); and 4) examine the relationship between RoB and effect estimates.

Methods: We conducted a systematic review of long-acting beta agonists (LABA) combined with inhaled corticosteroids (ICS) for adults with persistent asthma. Two reviewers independently assessed 107 trials using RoB, Jadad, and AC. One reviewer searched for study protocols. We assessed inter-rater agreement using weighted Kappa (κ) and the correlation between tools using Kendall's Tau (τ). Mean differences in effect sizes for RCTs with different RoB were calculated using inverse variance method and random effects model.

Results: Trials had good Jadad scores (median 4, IQR 3-4); however, 85% had unclear AC and 87% high RoB. The factor that most influenced RoB was the potential inappropriate influence of study sponsors (95% industry funded). Agreement on RoB domains was fair (κ = 0.40) to almost perfect (κ = 0.86), and moderate for overall RoB (κ = 0.41). Median time to complete RoB assessments was 21 minutes (IQR 14-27) and 12 minutes (IQR 9-16) to search for protocols. Protocols were identified for 5/42 studies (12%); in 3 cases the assessment of selective outcome reporting changed. There was low correlation between overall RoB vs. Jadad (τ = 0.04, p = 0.3) and AC (τ = -0.02, p = 0.7). Analyses comparing effect estimates and risk showed no important patterns.

Conclusions: Inter-rater agreement on RoB assessments was better than previously reported suggesting that review-specific guidelines are important. The correlation between RoB and Jadad was low suggesting measurement of different constructs (risk of bias vs. quality of reporting). The extensive involvement of the pharmaceutical industry in this LABA/ICS research should raise concerns about potential overestimates of treatment effects.

PubMed Disclaimer

Conflict of interest statement

Competing Interests: The authors have declared that no competing interests exist.

Figures

Figure 1
Figure 1. Mean differences in effect estimates for FEV1 across domains and overall risk of bias (with and without “other” sources of bias), high/unclear vs. low risk of bias (Note: TE = treatment effect; FEV1 = forced expiratory volume in 1 second.
Components were not estimable when all studies were rated the same [i.e., all high/unclear or all low risk of bias]).
Figure 2
Figure 2. Mean differences in effect estimates for FEV1 across domains and overall risk of bias (with and without “other” sources of bias), high vs. low/unclear risk of bias (Note: treatment effect; FEV1  =  forced expiratory volume in 1 second.
Components were not estimable when all studies were rated the same [i.e., all high or all low/unclear risk of bias]).
Figure 3
Figure 3. Mean differences in effect estimates for symptom-free days across domains and overall risk of bias (with and without “other” sources of bias), high/unclear vs. low risk of bias.
[Note: TE  =  treatment effect]).
Figure 4
Figure 4. Mean differences in effect estimates for symptom-free days across domains and overall risk of bias (with and without “other” sources of bias), high vs. low/unclear risk of bias.
(Note: TE  =  treatment effect. Components were not estimable when all studies were rated the same [i.e., all high or all low/unclear risk of bias]).
See this image and copyright information in PMC

References

    1. Higgins JPT, Green S. The Cochrane Collaboration; 2008. Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.0 [updated February 2008].
    1. Pildal J, Hrobjartsson A, Jorgensen KJ, Hilden J, Altman DG, et al. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol. 2007;36:847–57. - PubMed
    1. Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, et al. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ. 2009;339:b4012. - PMC - PubMed
    1. Bond K, Coyle D, O'Gorman K, Spooner C, Lemiere C, et al. [Technology report number 122]. Ottawa, Ontario, Canada, Canadian Agency for Drugs and Technologies in Health; 2009. Long-acting beta2-agonist and inhaled corticosteroid combination therapy for adult persistent asthma: systematic review of clinical outcomes and economic evaluation. - PubMed
    1. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996;17:1–12. - PubMed

MeSH terms