Oversampling and replacement strategies in propensity score matching: a critical review focused on small sample size in clinical settings

Abstract

Background: Propensity score matching is a statistical method that is often used to make inferences on the treatment effects in observational studies. In recent years, there has been widespread use of the technique in the cardiothoracic surgery literature to evaluate to potential benefits of new surgical therapies or procedures. However, the small sample size and the strong dependence of the treatment assignment on the baseline covariates that often characterize these studies make such an evaluation challenging from a statistical point of view. In such settings, the use of propensity score matching in combination with oversampling and replacement may provide a solution to these issues by increasing the initial sample size of the study and thus improving the statistical power that is needed to detect the effect of interest. In this study, we review the use of propensity score matching in combination with oversampling and replacement in small sample size settings.

Methods: We performed a series of Monte Carlo simulations to evaluate how the sample size, the proportion of treated, and the assignment mechanism affect the performances of the proposed approaches. We assessed the performances with overall balance, relative bias, root mean squared error and nominal coverage. Moreover, we illustrate the methods using a real case study from the cardiac surgery literature.

Results: Matching without replacement produced estimates with lower bias and better nominal coverage than matching with replacement when 1:1 matching was considered. In contrast to that, matching with replacement showed better balance, relative bias, and root mean squared error than matching without replacement for increasing levels of oversampling. The best nominal coverage was obtained by using the estimator that accounts for uncertainty in the matching procedure on sets of units obtained after matching with replacement.

Conclusions: The use of replacement provides the most reliable treatment effect estimates and that no more than 1 or 2 units from the control group should be matched to each treated observation. Moreover, the variance estimator that accounts for the uncertainty in the matching procedure should be used to estimate the treatment effect.

Keywords: Monte Carlo simulations; Oversampling; Propensity score matching; Replacement; Small samples.

Fig. 1

Balance assessment on the matched sets obtained in each scenario of the Monte Carlo simulations. The plot on the right shows the Average Standardized Mean Differences (ASMDs), and the left-graph the proportion of matched treated. On the x-axis, the level of oversampling is represented. Matching without and with replacement are identified by the colors. The shape of the dots distinguishes between weak and strong treatment assignment. The columns of the panel grids show the proportion of treated subjects in the dataset, whereas the rows show the sample size of the dataset

Fig. 2

Performances of the Average Treatment effect on the Treated (ATT) estimator on the matched sets obtained in each scenario of the primary set of Monte Carlo simulations. The top-left plot shows the relative bias, whereas the Root Mean Squared Error (RMSE) is shown in the top-right plot. On the bottom-left side, the 95 % Nominal Coverage (NC) obtained with the standard method is depicted, whereas on the bottom-right side the 95 % NC obtained with the Abadie-Imbens (AI) method is shown. On the x-axis, the level of oversampling is represented. Matching without and with replacement are identified by the colors. The shape of the dots distinguishes between weak and strong treatment assignment. The columns of the panel grids show the proportion of treated subjects in the dataset, whereas the rows show the sample size of the dataset

Fig. 3

Distributions of estimate Propensity Score (PS) in the unbalanced dataset of the case study. Colors identify the Jarvik2000 LVAD and the HeartWare HVAD groups, which were the treatment and control groups, respectively

Fig. 4

Estimates of the Average Treatment effect on the Treated (ATT), expressed as absolute risk reduction, in the matched sets of the case study obtained with all the evaluated PSM strategies. The dots represent the ATT estimates and the errorbars the 95 % CIs