The optimal dynamic treatment rule superlearner: considerations, performance, and application to criminal justice interventions

Abstract

The optimal dynamic treatment rule (ODTR) framework offers an approach for understanding which kinds of patients respond best to specific treatments - in other words, treatment effect heterogeneity. Recently, there has been a proliferation of methods for estimating the ODTR. One such method is an extension of the SuperLearner algorithm - an ensemble method to optimally combine candidate algorithms extensively used in prediction problems - to ODTRs. Following the ``causal roadmap," we causally and statistically define the ODTR and provide an introduction to estimating it using the ODTR SuperLearner. Additionally, we highlight practical choices when implementing the algorithm, including choice of candidate algorithms, metalearners to combine the candidates, and risk functions to select the best combination of algorithms. Using simulations, we illustrate how estimating the ODTR using this SuperLearner approach can uncover treatment effect heterogeneity more effectively than traditional approaches based on fitting a parametric regression of the outcome on the treatment, covariates and treatment-covariate interactions. We investigate the implications of choices in implementing an ODTR SuperLearner at various sample sizes. Our results show the advantages of: (1) including a combination of both flexible machine learning algorithms and simple parametric estimators in the library of candidate algorithms; (2) using an ensemble metalearner to combine candidates rather than selecting only the best-performing candidate; (3) using the mean outcome under the rule as a risk function. Finally, we apply the ODTR SuperLearner to the ``Interventions" study, an ongoing randomized controlled trial, to identify which justice-involved adults with mental illness benefit most from cognitive behavioral therapy to reduce criminal re-offending.

Keywords: causal roadmap; heterogeneous treatment effects; optimal dynamic treatment rules; precision health; superlearner.

Figure 1:

Performance of candidate estimators of the ODTR. Plot shows mean and 2.5th and 97.5th quantiles of the empirical mean of the true expected counterfactual outcome under the estimated ODTR, i.e., ${\mathbb{E}}_n\left[Q_0\left(Y|A=d_n^{*},W\right)\right]$ , an approximation of ${\mathbb{E}}_0\left[Q_0(Y|A=d_n^{*}(W),W)\right]$ , for DGP 1 (top two) and DGP 2 (bottom two). The horizontal black line depicts ${\mathbb{E}}_{P_{U,X}}\left[Y_{d_0^{*}}\right]$ ; red line depicts ${\mathbb{E}}_{P_{U,X}}[Y_1]$ ; blue line depicts ${\mathbb{E}}_{P_{U,X}}[Y_0]$ (where sometimes the blue and red lines coincide and thus overlap). We compare the ODTR SuperLearner to an incorrectly specified GLM (in gray, with N/A as the metalearner and a diamond with no fill). We also compare (1) having a SuperLearner library with (a) only algorithms that estimate the blip (i.e., “Blip only” libraries) that only have parametric algorithms (blue) or only have machine-learning blip algorithms (red) or both (purple) versus (b) an expanded or “Full” library with blip function regressions estimated via machine learning only (orange-yellow) or machine learning and parametric algorithms (green), with methods that directly estimate the ODTR and static rules, (2) having a metalearner (depicted on the x-axis) either that chooses one algorithm (i.e., the “discrete” SuperLearner) or combines blip predictions/treatment predictions (i.e., the “continuous” SuperLearner), and (3) using the MSE risk function (R _MSE as a square) versus the mean outcome under the candidate rule risk function ( $R_{E[Y_d]}$ as a triangle). The percent match at the top of each plot reports the average across simulation repetitions of the percent of the sample assigned their true optimal treatment by the estimated rule.

Figure 2:

Subgroup plots for each of the covariates for the “Interventions” data. The x-axis for each of the plots is the different levels of the covariates; the y-axis is the difference in proportion of people who were not re-arrested between those who received CBT versus TAU, in that covariate subgroup.

Figure 3:

Distribution of predicted blip estimates from the ODTR SuperLearner. The frequencies are divided into three groups because the ODTR SuperLearner allocated all coefficient weights to a GLM using substance use, a variable with only 3 treatment levels. One can interpret the ODTR SuperLearner for this sample as follows: CBT may reduce the probability of re-arrest among justice-involved adults with low levels of substance use. Estimation and inference of the value of the ODTR SuperLearner compared to, for example, treating everyone or no one, informs us if there is, in fact, a differential effect by substance use, and thus a benefit to assigning CBT in this individualized way.