Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Review
. 2011 May;8(5):e1001026.
doi: 10.1371/journal.pmed.1001026. Epub 2011 May 3.

Meta-analyses of adverse effects data derived from randomised controlled trials as compared to observational studies: methodological overview

Su Golder  1 , Yoon K Loke, Martin Bland
Affiliations
Review

Meta-analyses of adverse effects data derived from randomised controlled trials as compared to observational studies: methodological overview

Su Golder et al. PLoS Med. 2011 May.

Abstract

Background: There is considerable debate as to the relative merits of using randomised controlled trial (RCT) data as opposed to observational data in systematic reviews of adverse effects. This meta-analysis of meta-analyses aimed to assess the level of agreement or disagreement in the estimates of harm derived from meta-analysis of RCTs as compared to meta-analysis of observational studies.

Methods and findings: Searches were carried out in ten databases in addition to reference checking, contacting experts, citation searches, and hand-searching key journals, conference proceedings, and Web sites. Studies were included where a pooled relative measure of an adverse effect (odds ratio or risk ratio) from RCTs could be directly compared, using the ratio of odds ratios, with the pooled estimate for the same adverse effect arising from observational studies. Nineteen studies, yielding 58 meta-analyses, were identified for inclusion. The pooled ratio of odds ratios of RCTs compared to observational studies was estimated to be 1.03 (95% confidence interval 0.93-1.15). There was less discrepancy with larger studies. The symmetric funnel plot suggests that there is no consistent difference between risk estimates from meta-analysis of RCT data and those from meta-analysis of observational studies. In almost all instances, the estimates of harm from meta-analyses of the different study designs had 95% confidence intervals that overlapped (54/58, 93%). In terms of statistical significance, in nearly two-thirds (37/58, 64%), the results agreed (both studies showing a significant increase or significant decrease or both showing no significant difference). In only one meta-analysis about one adverse effect was there opposing statistical significance.

Conclusions: Empirical evidence from this overview indicates that there is no difference on average in the risk estimate of adverse effects of an intervention derived from meta-analyses of RCTs and meta-analyses of observational studies. This suggests that systematic reviews of adverse effects should not be restricted to specific study types. Please see later in the article for the Editors' Summary.

PubMed Disclaimer

Conflict of interest statement

The Academic Editor, Jan P. Vandenbroucke, has disclosed that he has worked with authors SG and YKL on another project unrelated to the study reported in this paper. He also discloses an intellectual competing interest in that he has previously published the theoretical viewpoint that estimates of harms outcomes in observational studies may be as valid as results of randomized trials, and may be more generalizable. The authors have no competing interests to declare.

Figures

Figure 1
Figure 1. Flow chart for included studies.
Figure 2
Figure 2. Meta-analysis of RORs from RCTs versus all observational studies.
Studies are listed by first author's last name and year of publication (Loke 2008 is ; AHRQ 2002 is [114]). In some studies more than one outcome or intervention was assessed. In these instances, indicated by the lowercase letters after the study year, the data were entered in the meta-analysis separately. Other studies compared RCTs to cohort studies and case-control studies separately and therefore are listed twice (with no lowercase letter after the study year).
Figure 3
Figure 3. Contour funnel plot: discrepancy (ln ROR) between study designs in relation to precision of estimates (1/SE).
See this image and copyright information in PMC

References

    1. Chou R, Helfand M. Challenges in systematic reviews that assess treatment harms. Ann Intern Med. 2005;142:1090–1099. - PubMed
    1. Mittmann N, Liu BA, Knowles SR, Shear NH. Meta-analysis and adverse drug reactions. CMAJ. 1999;160:987–988. - PMC - PubMed
    1. Ioannidis JP, Mulrow CD, Goodman SN. Adverse events: the more you search, the more you find. Ann Intern Med. 2006;144:298–300. - PubMed
    1. Levine M, Walter S, Lee H, Haines T, Holbrook A, et al. User's guides to the medical literature, IV: how to use an article about harm. JAMA. 1994;271:1615–1619. - PubMed
    1. Meade MO, Cook DJ, Kernerman P, Bernard G. How to use articles about harm: the relationship between high tidal volumes, ventilating pressures, and ventilator-induced lung injury. Crit Care Med. 1997;25:1915–1922. - PubMed