Heterogeneous treatment effect analysis based on machine-learning methodology

Abstract

Heterogeneous treatment effect (HTE) analysis focuses on examining varying treatment effects for individuals or subgroups in a population. For example, an HTE-informed understanding can critically guide physicians to individualize the medical treatment for a certain disease. However, HTE analysis has not been widely recognized and used, even given the explosive increase of data availability attributed to the arrival of the Big Data era. Part of the reason behind its underuse is that data are often of high dimension and high complexity, which pose significant challenges for applying conventional HTE analysis methods. To meet these challenges, a newly developed causal forest HTE method has been derived from the random forest machine-learning algorithm. We conducted a systematic performance evaluation for the causal forest method against the conventional two-step method by simulating scenarios with different levels of complexity for the analysis. Our results show that causal forest outperforms the conventional HTE method in assessing treatment effect, especially when data are complex (e.g., nonlinear) and high dimensional, suggesting that causal forest is a promising tool for real-world applications of HTE analysis.

FIGURE 1

(a) Homogenous treatment effect (no treatment heterogeneity): the outcome of the treatment shows variation across the individuals and between treatment groups, but the treatment effect (i.e., the difference of the outcomes depicted by the dotted lines between the two treatment outcome curves) is the same for every individual. (b) Heterogeneous treatment effect (HTE): treatment effect varies among individuals. Some individuals benefit more, some less, and some might not benefit at all from the treatment

FIGURE 2

Treatment effect in a data set where the relationship between the two covariates ×1 and treatment effect is nonlinear. The (a) “true” treatment effect with varying values of ×1 and ×2 and the predicted treatment effect using (b) causal forest and (c) the two‐step method. The treatment effect is denoted by color from blue (low) to red (high)

FIGURE 3

Comparison of the performance of the causal forest and two‐step methods. Results are based on 200 replicated simulations. Mean (bar height) and standard deviation (error bar) of the root mean square error (RMSE) are displayed

FIGURE 4

The variable importance determined by causal forest for high‐dimensional simulated data based on Model IV. The preset significant covariates are shown in orange. The five preset important covariates were identified, as their variable importance values are greater than the significance threshold (dashed). Please refer to Supplementary Information for a description of the statistical significance test

FIGURE 5

Incremental gains curves (or Qini curves) from each model. This curve shows the cumulative number of incremental individuals with positive treatment effect relative to the cumulative number of the targeted population. The dashed diagonal line depicts the theoretical incremental individuals with positive treatment effect from random targeting, whereas the gray line refers to the true treatment effect. For each mode, the incremental gain curves shown are the average of all the curves from 200 simulation replications. The Qini coefficients displayed on each panel are the average values from 200 simulation replications