Assessing the Impact of Context Inference Error and Partial Observability on RL Methods for Just-In-Time Adaptive Interventions

Abstract

Just-in-Time Adaptive Interventions (JITAIs) are a class of personalized health interventions developed within the behavioral science community. JITAIs aim to provide the right type and amount of support by iteratively selecting a sequence of intervention options from a pre-defined set of components in response to each individual's time varying state. In this work, we explore the application of reinforcement learning methods to the problem of learning intervention option selection policies. We study the effect of context inference error and partial observability on the ability to learn effective policies. Our results show that the propagation of uncertainty from context inferences is critical to improving intervention efficacy as context uncertainty increases, while policy gradient algorithms can provide remarkable robustness to partially observed behavioral state information.

Keywords: Reinforcement learning; adaptive interventions; context inference; empirical evaluation; mobile health; partial observability.

Figure 1:

Context inference error as a function of $\sigma$.

Figure 2:

(a) Effect of learning with most likely context and context probabilities for DQN. (b) Effect of learning with most likely context and context probabilities for REINFORCE. (c) Effect of learning with most likely contexts and partial observability for REINFORCE and DQN. (d) Effect of learning with context probabilities and partial observability for REINFORCE and DQN.

Figure 3:

The top row of plots shows the distribution of actions selected by REINFORCE when given access to context probabilities. The bottom row of plots shows the distribution of actions selected by REINFORCE when given access only to the inferred most likely context.

Figure 4:

Learning curves of DQN and REINFORCE.

Figure 5:

The top row of plots shows the distribution of actions selected by DQN when given access to context probabilities. The bottom row of plots shows the distribution of actions selected by DQN when given access only to the inferred most likely context.

Figure 6:

Performance as a function of the disengagement increment ${ϵ}_d$ and decay parameters ${\delta }_d$, for DQN (top row) and REINFORCE (bottom row).