Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Review
. 2021 Dec 6;19(1):307.
doi: 10.1186/s12916-021-02176-1.

Comparative effectiveness and safety of pharmaceuticals assessed in observational studies compared with randomized controlled trials

Yoon Duk Hong  1 , Jeroen P Jansen  2   3 , John Guerino  4 , Marc L Berger  5 , William Crown  6 , Wim G Goettsch  7   8 , C Daniel Mullins  9 , Richard J Willke  10 , Lucinda S Orsini  11
Affiliations
Review

Comparative effectiveness and safety of pharmaceuticals assessed in observational studies compared with randomized controlled trials

Yoon Duk Hong et al. BMC Med. .

Abstract

Background: There have been ongoing efforts to understand when and how data from observational studies can be applied to clinical and regulatory decision making. The objective of this review was to assess the comparability of relative treatment effects of pharmaceuticals from observational studies and randomized controlled trials (RCTs).

Methods: We searched PubMed and Embase for systematic literature reviews published between January 1, 1990, and January 31, 2020, that reported relative treatment effects of pharmaceuticals from both observational studies and RCTs. We extracted pooled relative effect estimates from observational studies and RCTs for each outcome, intervention-comparator, or indication assessed in the reviews. We calculated the ratio of the relative effect estimate from observational studies over that from RCTs, along with the corresponding 95% confidence interval (CI) for each pair of pooled RCT and observational study estimates, and we evaluated the consistency in relative treatment effects.

Results: Thirty systematic reviews across 7 therapeutic areas were identified from the literature. We analyzed 74 pairs of pooled relative effect estimates from RCTs and observational studies from 29 reviews. There was no statistically significant difference (based on the 95% CI) in relative effect estimates between RCTs and observational studies in 79.7% of pairs. There was an extreme difference (ratio < 0.7 or > 1.43) in 43.2% of pairs, and, in 17.6% of pairs, there was a significant difference and the estimates pointed in opposite directions.

Conclusions: Overall, our review shows that while there is no significant difference in the relative risk ratios between the majority of RCTs and observational studies compared, there is significant variation in about 20% of comparisons. The source of this variation should be the subject of further inquiry to elucidate how much of the variation is due to differences in patient populations versus biased estimates arising from issues with study design or analytical/statistical methods.

Keywords: Observational data; Pharmaceuticals; Real-world evidence.

PubMed Disclaimer

Conflict of interest statement

Yoon Duk Hong, John Guerino, Marc L. Berger, William Crown, Richard J. Willke, Wim G. Goettsch, and Lucinda S. Orsini have no conflicts of interest to report. Jeroen P. Jansen is a part-time employee of Precision Medicine Group (PMG) (PRECISIONheor) and has stock options from Precision Medicine Group. PMG provides contracted research services to pharmaceutical and biotech industry. C. Daniel Mullins has received consulting fees from AstraZeneca, Bayer, Incyte, Merck, Pfizer, and Takeda and has received support from Bayer and Pfizer for attending meetings and/or travel.

Figures

Fig. 1
Fig. 1
Diagram depicting literature screening process
Fig. 2
Fig. 2
Relative effect measures (RR, OR, HR) from observational studies (y-axis) versus corresponding relative effect measures from randomized controlled trials (x-axis) across 74 pairs of pooled relative treatment effects
See this image and copyright information in PMC

References

    1. Anglemyer A, Horvath HT, Bero L. Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials. Cochrane Database Syst Rev. 2014;4(4):MR000034. doi: 10.1002/14651858.MR000034.pub2. - DOI - PMC - PubMed
    1. Concato J, Shah N, Horwitz RI. Randomized controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000;342(25):1887–1892. doi: 10.1056/NEJM200006223422507. - DOI - PMC - PubMed
    1. Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med. 2000;342(25):1878–1886. doi: 10.1056/NEJM200006223422506. - DOI - PubMed
    1. Madigan D, Ryan P, Schuemie M, et al. Evaluating the impact of database heterogeneity on observational study results. Am J Epidemiol. 2013;178(4):645–651. doi: 10.1093/aje/kwt010. - DOI - PMC - PubMed
    1. Forbes S, Dahabreh I. Benchmarking observational analyses against randomized trials: a review of studies assessing propensity score methods. J Gen Intern Med. 2020;35(5):1396–1404. doi: 10.1007/s11606-020-05713-5. - DOI - PMC - PubMed

Substances