Q- and A-learning Methods for Estimating Optimal Dynamic Treatment Regimes

Abstract

In clinical practice, physicians make a series of treatment decisions over the course of a patient's disease based on his/her baseline and evolving characteristics. A dynamic treatment regime is a set of sequential decision rules that operationalizes this process. Each rule corresponds to a decision point and dictates the next treatment action based on the accrued information. Using existing data, a key goal is estimating the optimal regime, that, if followed by the patient population, would yield the most favorable outcome on average. Q- and A-learning are two main approaches for this purpose. We provide a detailed account of these methods, study their performance, and illustrate them using data from a depression study.

Keywords: Advantage learning; bias-variance tradeoff; model misspecification; personalized medicine; potential outcomes; sequential decision making.

Fig 1

Monte Carlo MSE ratios for estimators of components of ψ₁ (left and center panels) and efficiencies $R({\widehat{d}}_Q^{\text{opt}})\text{and}R({\widehat{d}}_A^{\text{opt}})$ for estimating the true d^opt (right panel) under misspecification of the propensity model. MSE ratios > 1 favor Q-learning

Fig 2

Monte Carlo MSE ratios for estimators of components of ψ₁ (left and center panels) and efficiencies $R({\widehat{d}}_Q^{\text{opt}})\text{and}R({\widehat{d}}_A^{\text{opt}})$ for estimating the true d^opt (right panel) under misspecification of the Q-function. MSE ratios > 1 favor Q-learning

Fig 3

Monte Carlo MSE ratios for estimators of components of ψ₁ (left and center panels) and efficiencies $R({\widehat{d}}_Q^{\text{opt}})\text{and}R({\widehat{d}}_A^{\text{opt}})$ for estimating the true d^opt (right panel) under misspecification of both the propensity model and the Q-function. MSE ratios > 1 favor Q-learning

Fig 4

Monte Carlo MSE ratios for estimators of components of ψ₂ and ψ₁ (upper row and lower row left and center panels) and efficiencies $R({\widehat{d}}_Q^{\text{opt}})\text{and}R({\widehat{d}}_A^{\text{opt}})$ for estimating the true d^opt (lower right panel) under misspecification of the propensity model. MSE ratios > 1 favor Q-learning

Fig 5

Monte Carlo MSE ratios for estimators of components ofψ₂ and ψ₁ (upper row and lower row left and center panels) and efficiencies $R({\widehat{d}}_Q^{\text{opt}})\text{and}R({\widehat{d}}_A^{\text{opt}})$ for estimating the true d^opt (lower right panel) under misspecification of the Q-functions. MSE ratios > 1 favor Q-learning

Fig 6

Monte Carlo MSE ratios for estimators of components of ψ₂ and ψ₁ (upper row and lower row left and center panels) and efficiencies $R({\widehat{d}}_Q^{\text{opt}})\text{and}R({\widehat{d}}_A^{\text{opt}})$ for estimating the true d^opt (lower right panel) under misspecification of both the propensity models and Q-functions. MSE ratios > 1 favor Q-learning