Modeling Avoidance in Mood and Anxiety Disorders Using Reinforcement Learning

Abstract

Background: Serious and debilitating symptoms of anxiety are the most common mental health problem worldwide, accounting for around 5% of all adult years lived with disability in the developed world. Avoidance behavior-avoiding social situations for fear of embarrassment, for instance-is a core feature of such anxiety. However, as for many other psychiatric symptoms the biological mechanisms underlying avoidance remain unclear.

Methods: Reinforcement learning models provide formal and testable characterizations of the mechanisms of decision making; here, we examine avoidance in these terms. A total of 101 healthy participants and individuals with mood and anxiety disorders completed an approach-avoidance go/no-go task under stress induced by threat of unpredictable shock.

Results: We show an increased reliance in the mood and anxiety group on a parameter of our reinforcement learning model that characterizes a prepotent (pavlovian) bias to withhold responding in the face of negative outcomes. This was particularly the case when the mood and anxiety group was under stress.

Conclusions: This formal description of avoidance within the reinforcement learning framework provides a new means of linking clinical symptoms with biophysically plausible models of neural circuitry and, as such, takes us closer to a mechanistic understanding of mood and anxiety disorders.

Keywords: Anxiety; Avoidance; Diathesis–stress; Pavlovian bias; Reinforcement learning; Threat of shock.

Figure 1

Experimental paradigm. The trial sequence for each trial-type condition under threat (red) and safe (blue) conditions. There were equal numbers of go to win, go to avoid, no-go to win reward, and no-go to avoid losing trials within each safe and threat block, and these were randomly ordered within each block (note that safe sequence proceeds in the same way as the threat sequence but is curtailed here for brevity). The prepotent pavlovian bias to a win is a go response (approach) and the prepotent pavlovian response to a loss is no-go (avoid); hence in go to win reward and no-go to avoid losing, the bias and task instructions are aligned, but in go to avoid losing and no-go to win reward participants have to learn to overcome their avoidance and approach biases, respectively. The safe and threat blocks were presented in alternating order, counterbalanced across participants. A different set of fractal cues was used for the safe and threat blocks, counterbalanced across participants. At feedback, a face (happy +10 points, fear –10 points) was shown 80% of the time, and no points (i.e., a yellow bar [not shown in the figure]) was shown 20% of the time.

Figure 2

Self-report anxiety and task performance. Between groups, (A) our mood and anxiety sample reported significantly higher trait anxiety scores (data missing for two participants in the control group [HC] [green] and one in the mood and anxiety group [ANX] [gray]), while (B) the whole sample reported increased (induced) anxiety, rated retrospectively, under threat relative to safe (Saf) conditions (violin plots; each point represents a subject, background shading represents estimated distribution). (C) Collapsed mean accuracy differs as a function of trial type, but this ignores that (D) performance on the task changed over time, such that the probability of making a response [P(go); as distinct from accuracy in panel (C)] differed as a function of trial type, condition, group, and time (shading represents SEM). Avo, avoid; Thr, threat.

Figure 3

Model fitting and comparison. Four different population distributions were tested separated by (A) group and threat condition (four distributions); (B) by threat condition alone (two distributions); (C) blind to group and threat condition (one distribution); and (D) by group alone (two distributions). Comparison of models and distributions using integrated Bayesian information criteria (iBIC) scores (colors match distributions throughout figure) revealed a winning model of standard + 2 approach-avoid + 2 learning rates, fit across a single prior distribution (inset zoomed in on the distribution comparison for this model). Box-and-whisker plots of the recovered parameters from the wining model/distribution are presented in panel (F) separated by group and condition (red triangles denote means, lines denote medians; based on individual parameter estimates). Log scales are used for the sensitivity and approach-avoidance parameters to aid visualization of these exponentially transformed parameters. ANX, mood and anxiety group; Ap-Av, approach avoid; Approach, approach bias; Avoid, avoidance bias; HC, healthy control group; LR, learning rate; Pun, punishment; Rew, reward; Sense, sensitivity; Stand, standard.

Figure 4

Posterior predictive model. Running the estimated parameters for each subject through a posterior predictive model recovered both (A) average go probabilities for each trial type (sensitivity plots: each marker represents one subject under one condition so there are twice as many markers as subjects) and (B) group-averaged trial-by-trial performance (compare to real data in Figure 2C). In panel (B) green shows healthy control group (HC) and gray shows mood and anxiety group (ANX). Comparing parameters across group and condition revealed (C) a significantly higher avoidance bias parameter in pathological anxiety across conditions as well as greater threat-potentiated avoidance in pathological anxiety (error bars represent SEM). Avo, avoid; Saf, safe; Thr, threat.