Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2011 Aug 16:343:d4909.
doi: 10.1136/bmj.d4909.

Inconsistency between direct and indirect comparisons of competing interventions: meta-epidemiological study

Fujian Song  1 , Tengbin Xiong, Sheetal Parekh-Bhurke, Yoon K Loke, Alex J Sutton, Alison J Eastwood, Richard Holland, Yen-Fu Chen, Anne-Marie Glenny, Jonathan J Deeks, Doug G Altman
Affiliations

Inconsistency between direct and indirect comparisons of competing interventions: meta-epidemiological study

Fujian Song et al. BMJ. .

Abstract

Objective: To investigate the agreement between direct and indirect comparisons of competing healthcare interventions.

Design: Meta-epidemiological study based on sample of meta-analyses of randomised controlled trials. Data sources Cochrane Database of Systematic Reviews and PubMed. Inclusion criteria Systematic reviews that provided sufficient data for both direct comparison and independent indirect comparisons of two interventions on the basis of a common comparator and in which the odds ratio could be used as the outcome statistic.

Main outcome measure: Inconsistency measured by the difference in the log odds ratio between the direct and indirect methods.

Results: The study included 112 independent trial networks (including 1552 trials with 478,775 patients in total) that allowed both direct and indirect comparison of two interventions. Indirect comparison had already been explicitly done in only 13 of the 85 Cochrane reviews included. The inconsistency between the direct and indirect comparison was statistically significant in 16 cases (14%, 95% confidence interval 9% to 22%). The statistically significant inconsistency was associated with fewer trials, subjectively assessed outcomes, and statistically significant effects of treatment in either direct or indirect comparisons. Owing to considerable inconsistency, many (14/39) of the statistically significant effects by direct comparison became non-significant when the direct and indirect estimates were combined.

Conclusions: Significant inconsistency between direct and indirect comparisons may be more prevalent than previously observed. Direct and indirect estimates should be combined in mixed treatment comparisons only after adequate assessment of the consistency of the evidence.

PubMed Disclaimer

Conflict of interest statement

Competing interests: All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: all authors had support from the UK Medical Research Council for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

Figures

None
Fig 1 Trial network used to investigate inconsistency between direct and indirect comparison. Each trial network consists of three sets of independent trials: one set for direct comparison of A versus B and two sets for adjusted indirect comparison of A versus B with C as common comparator.
None
Fig 2 Inconsistency between direct and indirect comparison: ratio of odds ratios (ROR) from 112 included trial networks. ROR=1 indicates no difference in odds ratios between direct and indirect comparison
None
Fig 3 One sided funnel plot of (absolute) estimated inconsistency between direct and indirect comparison. Dotted line indicates statistical significance at P=0.05
None
Fig 4 Inconsistency between direct and indirect comparisons of newer or more intense with older or less intense interventions. Data from 65 relevant trial networks. Ratio of odds ratios (ROR) <1 indicates that effect of newer or more intense interventions is greater in direct comparison than in indirect comparison
See this image and copyright information in PMC

References

    1. Glenny AM, Altman DG, Song F, Sakarovitch C, Deeks JJ, D’Amico R, et al. Indirect comparisons of competing interventions. Health Technol Assess 2005;9:1-134,iii-iv. - PubMed
    1. Edwards SJ, Clarke MJ, Wordsworth S, Borrill J. Indirect comparisons of treatments based on systematic reviews of randomised controlled trials. Int J Clin Pract 2009;63:841-54. - PubMed
    1. Song F, Loke YK, Walsh T, Glenny AM, Eastwood AJ, Altman DG. Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: a survey of published systematic reviews. BMJ 2009;338:b1147. - PMC - PubMed
    1. Donegan S, Williamson P, Gamble C, Tudur-Smith C. Indirect comparisons: a review of reporting and methodological quality. PLoS One 2010;5:e11054. - PMC - PubMed
    1. Bucher HC, Guyatt GH, Griffith LE, Walter SD. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epidemiol 1997;50:683-91. - PubMed

Publication types

LinkOut - more resources