Do observational studies using propensity score methods agree with randomized trials? A systematic comparison of studies on acute coronary syndromes

Abstract

Aims: Randomized controlled trials (RCTs) are the gold standard for assessing the efficacy of therapeutic interventions because randomization protects from biases inherent in observational studies. Propensity score (PS) methods, proposed as a potential solution to confounding of the treatment-outcome association, are widely used in observational studies of therapeutic interventions for acute coronary syndromes (ACS). We aimed to systematically assess agreement between observational studies using PS methods and RCTs on therapeutic interventions for ACS.

Methods and results: We searched for observational studies of interventions for ACS that used PS methods to estimate treatment effects on short- or long-term mortality. Using a standardized algorithm, we matched observational studies to RCTs based on patients' characteristics, interventions, and outcomes ('topics'), and we compared estimates of treatment effect between the two designs. When multiple observational studies or RCTs were identified for the same topic, we performed a meta-analysis and used the summary relative risk for comparisons. We matched 21 observational studies investigating 17 distinct clinical topics to 63 RCTs (median = 3 RCTs per observational study) for short-term (7 topics) and long-term (10 topics) mortality. Estimates from PS analyses differed statistically significantly from randomized evidence in two instances; however, observational studies reported more extreme beneficial treatment effects compared with RCTs in 13 of 17 instances (P = 0.049). Sensitivity analyses limited to large RCTs, and using alternative meta-analysis models yielded similar results.

Conclusion: For the treatment of ACS, observational studies using PS methods produce treatment effect estimates that are of more extreme magnitude compared with those from RCTs, although the differences are rarely statistically significant.

Figure 1

Search strategy flow. ACS, acute coronary syndromes; CAD, coronary artery disease; PS, propensity score; RCTs, randomized controlled trials.

Figure 2

(A) Relative risks from randomized controlled trials (white diamonds) and observational studies using propensity score methods (red squares), reporting on long-term mortality. When more than two randomized or observational studies were available for a topic, estimates were obtained from random-effects meta-analysis. (B) Corresponding relative effect sizes (black squares) comparing the effect sizes from randomized and propensity-based observational studies on long-term mortality. In both panels, extending lines denote 95% confidence intervals of estimates. See Supplementary material online, Table S1 for detailed study-level information. Please see Table 1 for abbreviation definitions.

Figure 3

(A) Relative risks from randomized controlled trials (white diamonds) and observational studies using propensity score methods (red squares), reporting on short-term mortality. (B) Corresponding relative effect sizes (black squares) comparing the relative risks from randomized trials and observational studies using propensity score methods. Layout is similar to Figure 2. Arrows indicate lower or upper bounds of confidence intervals that were outside the plotted range of values. See Supplementary material online, Table S1 for detailed study-level information. Please see Table 1 for abbreviation definitions.