Rational thoughts in neural codes

Abstract

Complex behaviors are often driven by an internal model, which integrates sensory information over time and facilitates long-term planning to reach subjective goals. A fundamental challenge in neuroscience is, How can we use behavior and neural activity to understand this internal model and its dynamic latent variables? Here we interpret behavioral data by assuming an agent behaves rationally—that is, it takes actions that optimize its subjective reward according to its understanding of the task and its relevant causal variables. We apply a method, inverse rational control (IRC), to learn an agent’s internal model and reward function by maximizing the likelihood of its measured sensory observations and actions. This thereby extracts rational and interpretable thoughts of the agent from its behavior. We also provide a framework for interpreting encoding, recoding, and decoding of neural data in light of this rational model for behavior. When applied to behavioral and neural data from simulated agents performing suboptimally on a naturalistic foraging task, this method successfully recovers their internal model and reward function, as well as the Markovian computational dynamics within the neural manifold that represent the task. This work lays a foundation for discovering how the brain represents and computes with dynamic latent variables.

Keywords: cognition; computation; neural coding; neuroscience; rational.

Fig. 1.

(A and B) Graphical model of a POMDP (A) and the IRC problem (B). Open circles denote latent variables, and solid circles denote observable variables. For the POMDP, the agent knows its beliefs but must infer the world state. For IRC, the scientist knows the world state but must infer the beliefs. The real-world dynamics depend on parameters $\varphi$, while the belief dynamics and actions of the agent depend on parameters $\theta$, which include both its assumptions about the stochastic world dynamics and observations and its own subjective rewards and costs.

Fig. 2.

Illustration of foraging task with latent dynamics and partially observable sensory data. The reward availability in each of the two boxes evolves according to a telegraph process, switching between available (red) and unavailable (blue), and colors give the animal an ambiguous sensory cue about the reward availability. The agent may travel between the locations of the two boxes. When a button is pushed to open a box, the agent receives any available reward.

Fig. 3.

Successful recovery of agent model by inverse rational control. The agent was a neural network trained to imitate a suboptimal but rational teacher and tested on a novel task. (A) The estimated parameters converge to the optimal point of the observed data log-likelihood (white star). Since the parameter space is high-dimensional, we project it onto the first two principal components $\mathbf{u},\mathbf{v}$ of the learning trajectory for $\theta$ (blue). The estimated parameters differ slightly from the teacher’s parameters (green dot) due to data limitations. (B) Comparison of the teacher’s parameters and the estimated parameters. Error bars show 95% confidence intervals (CI) based on the Hessian of log-likelihood (SI Appendix, Fig. S2). (C) Estimated and the teacher’s true marginal belief dynamics over latent reward availability. These estimates are informed by the noisy color data at each box and the times and locations of the agent’s actions. The posteriors over beliefs are consistent with the dynamics of the teacher’s beliefs (blue line). (D) Teacher’s beliefs versus IRC belief posteriors averaged over all times when the teacher had the same beliefs, $\overline{p}=⟨p\left({\widehat{b}}_t|a_{1:T},o_{1:T}\right)|b_t⟩$. These mean posteriors $\overline{p}$ concentrate around the true beliefs of the teacher. (E–H) Inferred distributions of (E) actions, (F) residence times, (G) intervals between consecutive button-presses, and (H) intervals between movements.

Fig. 4.

Schematic for analyzing a dynamic neural code. (A) Graphical model of a POMDP problem with a solution implemented by neurons implicitly encoding beliefs. (B) We find how behaviorally relevant variables (here, beliefs) are encoded in measured neural activity through the function ${\check{b}}_t={\phi }_{\mathrm{e}\mathrm{n}\mathrm{c}}\left({\mathbf{r}}_t\right)$. (C) We then test our hypothesis that the brain recodes its beliefs rationally by testing whether the dynamics of the behaviorally estimated belief $\widehat{b}$ match the dynamics of the neurally estimated beliefs $\check{b}$, as expressed through the update dynamics ${\widehat{f}}_{\mathrm{d}\mathrm{y}\mathrm{n}}\left({\widehat{b}}_t,o_t\right)$ and recoding function ${\check{f}}_{\mathrm{r}\mathrm{e}\mathrm{c}}\left({\check{b}}_t,o_t\right)$. (D) Similarly, we test whether the brain decodes its beliefs rationally by comparing the behaviorally and neurally derived policies ${\widehat{\pi }}_{\mathrm{a}\mathrm{c}\mathrm{t}}$ and ${\check{\pi }}_{\mathrm{d}\mathrm{e}\mathrm{c}}$. Quantities estimated from behavior or from neurons are denoted by up-pointing or down-pointing hats, $\widehat{}$ and $\stackrel{ˇ}{}$, respectively (SI Appendix, Table S1).

Fig. 5.

Conceptual illustration of encoding and recoding. (A) Neural responses $\mathbf{r}$ inhabit a manifold (blue volume, here three-dimensional) embedded in the high-dimensional space of all possible neural responses. A neural encoding model divides this manifold into task-relevant and -irrelevant coordinates (blue and purple axes). We must estimate these coordinates from training data, given some inferred task-relevant targets $\mathbf{b}$. According to this encoding, many activity patterns $\mathbf{r}$ can correspond to the same vector of task variables $\mathbf{b}$. Any particular neural trajectory (white curve) is just one of many that would trace out the same task-relevant projection $\mathbf{b}\left(t\right)$ (black curves). The set of all neural activities consistent with one task-relevant trajectory therefore spans a manifold (gray ribbon). (B) After fitting an estimator of the task variables using training data, we can measure how well the encoding describes the task variables in a new testing dataset. Different encodings (red and green volumes) divide the same neural manifold differently into relevant and irrelevant coordinates, and the task variables $\check{\mathbf{b}}$ estimated from these neural encodings (red and green curves) will deviate in different ways from the variables $\widehat{\mathbf{b}}$ inferred from behavior (black). (C) The testing error of these neurally derived task variables (red, green) will be larger than the training error (blue). Task-relevant variables $\check{\mathbf{b}}$ derived from different encoding models may have the same total errors, but may nonetheless have different recoding dynamics. Here the smoother green dynamics are closer to the behaviorally inferred dynamics than the rougher red dynamics, which implies that these task-relevant dimensions better capture the computations implied by inverse rational control. SI Appendix, Fig. S3 provides more detail of good and bad recodings.

Fig. 6.

Analysis of neural coding of rational thoughts. (A) Encoding: Neurally derived beliefs $\check{b}$ match behaviorally derived beliefs $\widehat{b}$ based on IRC. Cross-validated neural beliefs are estimated from testing neural responses $\mathbf{r}$ using a linear estimator, $\check{b}=W\mathbf{r}+c$, with the weight matrix fitted from separate training data. (B) Recoding: Belief updates $\mathrm{\Delta }{\check{b}}_t$ from the neural recoding function match the corresponding belief updates from the task dynamics $\mathrm{\Delta }{\widehat{b}}_t$. Neural updates are estimated using nonlinear regression with radial basis functions (Materials and Methods). (C) Decoding: The policy ${\check{\pi }}_{\mathrm{d}\mathrm{e}\mathrm{c}}$ predicted by decoding neural beliefs approximately matches the policy ${\widehat{\pi }}_{\mathrm{a}\mathrm{c}\mathrm{t}}$ estimated from behavior by IRC. Neural policy is estimated from actions $a$ and neural beliefs $\check{b}$ using nonlinear multinomial regression (Materials and Methods).