Reinforcement-Learning-Informed Queries Guide Behavioral Change

Abstract

Algorithmically defined aspects of reinforcement learning correlate with psychopathology symptoms and change with symptom improvement following cognitive-behavioral therapy (CBT). Separate work in nonclinical samples has shown that varying the structure and statistics of task environments can change learning. Here, we combine these literatures, drawing on CBT-based guided restructuring of thought processes and computationally defined mechanistic targets identified by reinforcement-learning models in depression, to test whether and how verbal queries affect learning processes. Using a parallel-arm design, we tested 1,299 online participants completing a probabilistic reward-learning task while receiving repeated queries about the task environment (11 learning-query arms and one active control arm). Querying participants about reinforcement-learning-related task components altered computational-model-defined learning parameters in directions specific to the target of the query. These effects on learning parameters were consistent across depression-symptom severity, suggesting new learning-based strategies and therapeutic targets for evoking symptom change in mood psychopathology.

Keywords: depression; experimental therapeutics; psychopathology; reinforcement learning.

Fig. 1.

Task design and ratings. (a) A single trial with query in the guided learning task. Participants were presented two stimuli, made a choice, received the outcome for that choice, and on every third trial were then queried about a specific aspect of a stimulus—here, the least ever received for the last chosen option. (b) Participants’ ratings of engagement (first column), interest (second column), and difficulty (third column); ratings were obtained from a Likert-type scale where 0 = not at all and 10 = very much. The first row displays histograms of ratings for all participants, the second row displays relationships between ratings and changes in positive affect during the task, and the third row displays relationships between ratings and changes in negative affect. Each plot in the second and third rows displays a dot per participant, a line of best fit, and Pearson correlation coefficient.

Fig. 2.

Learning curves by query arm. Query arms are indicated by color/and or number. (a) Each panel shows learning curves and query accuracy over time for the indicated query arms. The active control arm is plotted in gray on each panel. The top half of each panel shows a running average (last five trials) of choice accuracy (y-axis), defined as the percentage of time the stimulus more likely to lead to a higher monetary outcome is chosen. Each bottom panel shows a running average (last three queries) of query accuracy (y-axis), defined as the signed difference between the actual and correct responses to queries. Here, 0 on the y-axis indicates perfect accuracy, and distance above or below 0 indicates greater query inaccuracy. The x-axis shows the trial number. (b) Overall choice accuracy by query arm. Bars indicate SEM. (c) Overall proportion of switches between options by query arm. Bars indicate SEM. For display purposes, query arm data are grouped by query target or stimulus chosen.

Fig. 3.

Changes in learning parameters by query arm relative to active control. Shaded areas represent the posterior distribution of the difference from active control for each query arm. Opaque colors represent distributions significantly different from 0 (95% of posterior above or below 0).

Fig. 4.

Relationships among choice accuracy, query accuracy, and symptom severity. On each plot, colored lines and dots indicate query arm (see Table 1). Each dot represents one participant, and colored lines represent the line of best fit per arm. Thick black lines indicate the line of best fit for all participants across all arms. “Depression,” “anxiety,” and “stress severity” refer to Depression Anxiety and Stress Scale depression, anxiety, and stress subscale scores, respectively. (a) Relationship between query and choice accuracy. Across all conditions, more accurate responses on queries (lower distance from correct value) were related to better choice accuracy. (b) Relationship between symptom severity and choice accuracy. Overall, higher symptom severity was related to worse choice accuracy. (c) Relationship between symptom severity and query accuracy. Overall, higher symptom severity was related to worse query accuracy.