Treatment Effects in Randomized and Nonrandomized Studies of Pharmacological Interventions: A Meta-Analysis

Abstract

Importance: Randomized clinical trials (RCTs) are widely regarded as the methodological benchmark for assessing clinical efficacy and safety of health interventions. There is growing interest in using nonrandomized studies to assess efficacy and safety of new drugs.

Objective: To determine how treatment effects for the same drug compare when evaluated in nonrandomized vs randomized studies.

Data sources: Meta-analyses published between 2009 and 2018 were identified in MEDLINE via PubMed and the Cochrane Database of Systematic Reviews. Data analysis was conducted from October 2019 to July 2024.

Study selection: Meta-analyses of pharmacological interventions were eligible for inclusion if both randomized and nonrandomized studies contributed to a single meta-analytic estimate.

Data extraction and synthesis: For this meta-analysis using a meta-epidemiological framework, separate summary effect size estimates were calculated for nonrandomized and randomized studies within each meta-analysis using a random-effects model and then these estimates were compared. The reporting of this study followed the Guidelines for Reporting Meta-Epidemiological Methodology Research and relevant portions of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) reporting guideline.

Main outcome and measures: The primary outcome was discrepancies in treatment effects obtained from nonrandomized and randomized studies, as measured by the proportion of meta-analyses where the 2 study types disagreed about the direction or magnitude of effect, disagreed beyond chance about the effect size estimate, and the summary ratio of odds ratios (ROR) obtained from nonrandomized vs randomized studies combined across all meta-analyses.

Results: A total of 346 meta-analyses with 2746 studies were included. Statistical conclusions about drug benefits and harms were different for 130 of 346 meta-analyses (37.6%) when focusing solely on either nonrandomized or randomized studies. Disagreements were beyond chance for 54 meta-analyses (15.6%). Across all meta-analyses, there was no strong evidence of consistent differences in treatment effects obtained from nonrandomized vs randomized studies (summary ROR, 0.95; 95% credible interval [CrI], 0.89-1.02). Compared with experimental nonrandomized studies, randomized studies produced on average a 19% smaller treatment effect (ROR, 0.81; 95% CrI, 0.68-0.97). There was increased heterogeneity in effect size estimates obtained from nonrandomized compared with randomized studies.

Conclusions and relevance: In this meta-analysis of treatment effects of pharmacological interventions obtained from randomized and nonrandomized studies, there was no overall difference in effect size estimates between study types on average, but nonrandomized studies both overestimated and underestimated treatment effects observed in randomized studies and introduced additional uncertainty. These findings suggest that relying on nonrandomized studies as substitutes for RCTs may introduce additional uncertainty about the therapeutic effects of new drugs.

Figure 1.. Flowchart of Selection of Meta-Analyses for Meta-Epidemiological Study

RCT indicates randomized clinical trial.

Figure 2.. Agreement of Summary Effect Size Estimates Obtained From Randomized and Nonrandomized Studies for 346 Clinical Questions

Each circle shows the summary odds ratio (OR) obtained from a meta-analysis of randomized clinical trials (RCTs; vertical axis) and nonrandomized studies (NRSs; horizonal axis) for 1 clinical question. An OR less than 1 indicates a beneficial effect. The solid orange line indicates perfect agreement (exact same summary OR obtained from randomized and nonrandomized studies) and the dashed orange lines indicate substantial disagreement (OR obtained from randomized studies is at most one-half of the OR obtained from nonrandomized studies, or vice versa). Results for alternative cutoff values for substantial disagreement are provided in eTable 2 in Supplement 1. Circles in the upper left quadrant show meta-analyses where NRS evidence indicates a beneficial effect (summary OR <1) and RCT evidence a detrimental effect (summary OR >1), and circles in the bottom right quadrant show meta-analyses where NRS evidence indicates a detrimental effect (summary OR >1) and RCT evidence a beneficial effect (summary OR <1). Circles in the upper right quadrant show meta-analyses where both NRS and RCT evidence indicate a detrimental effect; circles above the solid orange line indicate a larger detrimental effect size in RCTs and circles below the solid orange line indicate a larger detrimental effect size in NRSs. Circles in the bottom left quadrant show meta-analyses where both NRS and RCT evidence indicate a beneficial effect; circles above the solid orange line indicate a larger beneficial effect size in NRS and circles below the solid orange line indicate a larger beneficial effect size in RCTs.

Figure 3.. Discrepancies in Statistical Conclusions About Therapeutic Benefit of Pharmacological Interventions Based on Evidence Obtained From Nonrandomized Studies (NRSs) or Randomized Clinical Trials (RCTs)

The figure shows proportions of meta-analyses based on the statistical conclusions about the existence of a therapeutic benefit drawn from NRS or RCT evidence. A favorable or detrimental effect was deemed to exist if the 95% CI of the summary odds ratio did not include 1. Evidence was considered inconclusive if the 95% CI of the summary odds ratio included 1.

Figure 4.. Results of Meta–Meta-Analytic Comparison

The figure shows the ratio of odds ratios (ROR) comparing effect size estimates obtained from nonrandomized studies (NRSs) with effect size estimates obtained from randomized clinical trials (RCTs) and heterogeneity parameters (φ, between–meta-analysis heterogeneity; κ, increase in within–meta-analysis heterogeneity). Results are shown for all meta-analyses, followed by subgroup analyses by type of NRS, different types of outcomes, types of comparators, matching quality of RCTs and NRSs in the same meta-analysis, and high-quality publications. MA indicates meta-analyses.