A tutorial comparing different covariate balancing methods with an application evaluating the causal effects of substance use treatment programs for adolescents

Abstract

Randomized controlled trials are the gold standard for measuring causal effects. However, they are often not always feasible, and causal treatment effects must be estimated from observational data. Observational studies do not allow robust conclusions about causal relationships unless statistical techniques account for the imbalance of pretreatment confounders across groups and key assumptions hold. Propensity score and balance weighting (PSBW) are useful techniques that aim to reduce the observed imbalances between treatment groups by weighting the groups to look alike on the observed confounders. Notably, there are many methods available to estimate PSBW. However, it is unclear a priori which will achieve the best trade-off between covariate balance and effective sample size for a given application. Moreover, it is critical to assess the validity of key assumptions required for robust estimation of the needed treatment effects, including the overlap and no unmeasured confounding assumptions. We present a step-by-step guide to the use of PSBW for estimation of causal treatment effects that includes steps on how to evaluate overlap before the analysis, obtain estimates of PSBW using multiple methods and select the optimal one, check for covariate balance on multiple metrics, and assess sensitivity of findings (both the estimated treatment effect and statistical significance) to unobserved confounding. We illustrate the key steps using a case study examining the relative effectiveness of substance use treatment programs and provide a user-friendly Shiny application that can implement the proposed steps for any application with binary treatments.

Keywords: Balancing weights; Causal treatment effect; Propensity score; Sensitivity analysis; Unmeasured confounding.

Fig. 1

Density plot of control and treatment group, where there is lack of overlap in the support of the two groups. The data used for this plot are artificial to demonstrate the issue

Fig. 2

Density plots of Age, Substance Frequency Scale and Depressive Symptom Scale — light red is the control group (A-CRA) and with light blue the treatment group (MET/CBT5)

Fig. 3

The x-axis indicates the SMD of potential confounders, and the y-axis the correlation with the outcome. The black solid contours represent the size of the treatment effect, while the red dashed ones represent the p-value cut-offs (0.01, 0.05, 0.1 levels of significance). The blue dots represent the association of the confounders with the treatment and the outcome