Bias associated with using the estimated propensity score as a regression covariate

Abstract

The use of propensity score methods to adjust for selection bias in observational studies has become increasingly popular in public health and medical research. A substantial portion of studies using propensity score adjustment treat the propensity score as a conventional regression predictor. Through a Monte Carlo simulation study, Austin and colleagues. investigated the bias associated with treatment effect estimation when the propensity score is used as a covariate in nonlinear regression models, such as logistic regression and Cox proportional hazards models. We show that the bias exists even in a linear regression model when the estimated propensity score is used and derive the explicit form of the bias. We also conduct an extensive simulation study to compare the performance of such covariate adjustment with propensity score stratification, propensity score matching, inverse probability of treatment weighted method, and nonparametric functional estimation using splines. The simulation scenarios are designed to reflect real data analysis practice. Instead of specifying a known parametric propensity score model, we generate the data by considering various degrees of overlap of the covariate distributions between treated and control groups. Propensity score matching excels when the treated group is contained within a larger control pool, while the model-based adjustment may have an edge when treated and control groups do not have too much overlap. Overall, adjusting for the propensity score through stratification or matching followed by regression or using splines, appears to be a good practical strategy.

Keywords: matching; observational studies; stratification; weighting.

Figure 1

Density plots for the three distributions of X with 2000 control (black) and 500 treated observations (gray).

Figure 2

Absolute relative bias (y-axis) of treatment effect estimates by propensity score estimation method (x-axis) for linear, logistic, Poisson, and Cox regression models and each data distribution scenario (points on plot). A, no adjustment; B, ANCOVA; C, stratification; D, stratification with regression; E, spline; F, IPTW; G, IPTW and regression; H, optimal matching; I, optimal matching and regression). ○ Contained, △ common support, + some overlap.

Figure 3

Absolute relative bias (y-axis) of treatment effect estimates by propensity score estimation method (x-axis) for a linear regression model with five covariates and two data distribution scenarios (points on plot). A, no adjustment; B, ANCOVA; C, stratification; D, stratification with regression; E, spline; F, IPTW; G, IPTW and regression; H, optimal matching; I, optimal matching and regression. ○ Contained, + some overlap.