Different methods of balancing covariates leading to different effect estimates in the presence of effect modification

Abstract

A number of covariate-balancing methods, based on the propensity score, are widely used to estimate treatment effects in observational studies. If the treatment effect varies with the propensity score, however, different methods can give very different answers. The authors illustrate this effect by using data from a United Kingdom-based registry of subjects treated with anti-tumor necrosis factor drugs for rheumatoid arthritis. Estimates of the effect of these drugs on mortality varied from a relative risk of 0.4 (95% confidence interval: 0.16, 0.91) to a relative risk of 1.3 (95% confidence interval: 0.8, 2.25), depending on the balancing method chosen. The authors show that these differences were due to a combination of an interaction between propensity score and treatment effect and to differences in weighting subjects with different propensity scores. Thus, the methods are being used to calculate average treatment effects in populations with very different distributions of effect-modifying variables, resulting in different overall estimates. This phenomenon highlights the importance of careful selection of the covariate-balancing method so that the overall estimate has a meaningful interpretation.

Figure 1.

Effect of different methods of balancing covariates on actual balance of covariates between exposed and unexposed subjects. None, no balancing; Strat, stratification (5 strata); Wt. (All), weighting to the distribution of covariates in the entire sample; Wt. (Trt.), weighting to the distribution of covariates in the exposed subsample; M. (Rep), matching with replacement; M. (No Rep), matching without replacement; RR, rate ratio. Shaded boxes, interquartile range; white bars, median; whiskers, the most extreme observation less than 1.5 times the length of the shaded box beyond the shaded box; dots, individual observations beyond the whiskers; horizontal lines, relative change of ±5%, ±10%.

Figure 2.

Distribution of the linear predictor of the propensity score among exposed and unexposed subjects, without adjustment. ———, untreated; – – – , treated.

Figure 3.

Distribution of the linear predictor of the propensity score among exposed and unexposed subjects, after adjustment. ———, untreated, matched with replacement; – – – –, treated, matched with replacement; – ·· – ··, untreated, matched without replacement; ········, treated, matched without replacement.