A computational model for learning from repeated traumatic experiences under uncertainty

Abstract

Traumatic events can lead to lifelong, inflexible adaptations in threat perception and behavior, which characterize posttraumatic stress disorder (PTSD). This process involves associations between sensory cues and internal states of threat and then generalization of the threat responses to previously neutral cues. However, most formulations neglect adaptations to threat that are not specific to those associations. To incorporate nonassociative responses to threat, we propose a computational theory of PTSD based on adaptation to the frequency of traumatic events by using a reinforcement learning momentum model. Recent threat prediction errors generate momentum that influences subsequent threat perception in novel contexts. This model fits primary data acquired from a mouse model of PTSD, in which unpredictable footshocks in one context accelerate threat learning in a novel context. The theory is consistent with epidemiological data that show that PTSD incidence increases with the number of traumatic events, as well as the disproportionate impact of early life trauma. Because the theory proposes that PTSD relates to the average of recent threat prediction errors rather than the strength of a specific association, it makes novel predictions for the treatment of PTSD.

Keywords: Bayesian; Computational modeling; PTSD; Reinforcement learning; Stress sensitization; Trauma; Uncertainty.

Figure 1 –

David Marr’s Levels of Analysis for computational neuroscience as applied to PTSD. (A) Definition of the three levels of analysis from ref. 7. (B) Application of those levels to associative learning (left) and non-associative learning (right) in PTSD. (left) Associative learning is a well-characterized system with a clear computational goal of ethological relevance (Computational), a mathematically defined formal model (Algorithm), and neural circuit mechanisms (Implementation). (right) Non-associative learning is less well-understood but in this study conceptualizes linkages between non-associative Bayesian estimation of trauma and RL models.

Figure 2 –

A Bayesian observer can measure the rate of traumatic attacks more easily than lethality. (A) Schematic of a simplified doubly stochastic model of attacks (i.e., traumatic events). Attacks occur stochastically at each timepoint with a fixed probability $p_a$ . Conditional on attacks occurring, agents die with probability $p_d$. If agents survive, they estimate the ongoing probability of attacks according to Bayes’ rule. (B) Agents must estimate $p_a$ and $p_d$ in an information-poor environment. The number of attacks experienced by the typical agent is low, usually 3–5 over the course of a lifetime for $p_a=0.2$, a typical value for the lethality of predator attacks (30). (C) The Bayesian estimator of $p_a$ and $p_d$ for a typical example sequence of attacks shows tight convergence for $p_a$ and non-convergence for $p_d$. The wider spread in $p_d$ is indicative of the inability of the Bayesian agent to estimate the lethality of attacks. (D) As the agent continues over its lifetime (red to blue map), the estimate of $p_a$ slowly narrows (vertical lines, 95% intervals). Greater time allows the agent to accumulate greater evidence about the true value of $p_a$.

Figure 3 –

Early life traumas have a disproportionate effect on the estimated attack rate. (A) Characteristic examples of two distributions of attack frequencies. In the random attack model (uniform attack probability), attacks are uniformly distributed across the lifespan. In the early life stress model, an identical number of attacks are uniformly distributed across the first half of the lifespan. (B) Two Bayesian agents (one for early life stress, one for random attack) posterior distributions for attack rate sequentially measured across the lifespan, for the random and early attack models. The discrepancy between estimated and true attack rate (p_a=0.01) is greatest at the start of life due to a higher density of attacks in the early life stress model. Over the course of the lifespan, these two models converge.

Figure 4 –

Reinforcement learning with momentum allows improved estimation of autocorrelated attack rates. (A) Single traumatic events occur in different environmental states (contexts), leading to increased associated threat according to the RL model. (B) In the RL momentum model, the same series of attacks produces momentum which couples threat across contexts. Context C threat is due to momentum since the animal receives no footshocks in that state. (C) The momentum learning rate term of the RL momentum model enables extraction of information about fluctuating attack rates. Autoregressive attack rates were produced as shown in figure 3 to produce n=10000 simulated attack sequences (light blue, highest autoregression to dark blue, lowest autoregression). All attacks occur in a different context. In the absence of momentum, the agent cannot extract information about fluctuations in underlying attack rate. With higher momentum, the agent can extract information about the underlying attack rate fluctuations.

Figure 5 –

RL momentum fits threat behavioral data in a mouse model of PTSD. (A) Example mouse behavioral data across three days of in the stress-enhanced fear learning model of PTSD (upper), along with RL momentum fit to behavioral data (lower). (upper left) Freezing across 90 minutes (red) of exposure to 15 unpredictable footshocks (black; 1mA, 1s). (upper center) Freezing across subsequent exposure to 1 uncued footshock in a new context. (upper right) Freezing during re-test in the new context (lower left) Threat according to maximum likelihood model fit of the RL momentum model (threat associated with context A – blue, context B- green) on day 1, (lower center) day 6, and (lower right) day 7. (B-C) Averaged freezing data across (n=17 stressed, n=18 controls) on days 6 (B) and 7 (C). (D) Model comparison between classic RL model and RL momentum model for SEFL mice (n=17 stressed, n=18 controls). Bayes information criterion (BIC) was calculated (see Methods) for maximum likelihood fits of the RL model and RL momentum model for either unstressed animals (0 shocks on day 1) or stressed animals (15 shocks on day 1). Difference in BIC between the two models is shown for individual animals (gray dots; black dot for example data from (A)), mean BIC difference per condition as bars (blue – unstressed, pink – stressed).

Figure 6 –

RL momentum model offers a new perspective on mechanisms of extinction and symptom exacerbation in PTSD. (A) RL model: Two traumatic events in an initial context (context A; blue highlight) produce threat learning associated with that context (blue line) but no threat associated with a novel context (context B; green line) during exposure to that context (green highlights). Extinction occurs when exposure to the initial context A after the traumatic events causes threat prediction errors which decrease threat associated with context A (blue highlights, second and third exposures). (B) RL momentum model: Two traumatic events in initial context produce a momentum which increases threat in a novel context (green line). Re-exposure to initial threat context (context A; blue highlights) reduces threat associated with context A (blue line) but also reduces threat momentum (green line). Green dotted line shows counterfactual threat momentum if no re-exposure to context A had occurred). (C) RL momentum model demonstrates a novel explanation for relapse during exposure therapy. Exposure to smaller stressors (small lines) in a novel context increases threat associated with context B (green line) but also, via the momentum term, increases threat associated with the initial traumatic context A (blue line).