Accounting for multiscale processing in adaptive real-world decision-making via the hippocampus

Abstract

For adaptive real-time behavior in real-world contexts, the brain needs to allow past information over multiple timescales to influence current processing for making choices that create the best outcome as a person goes about making choices in their everyday life. The neuroeconomics literature on value-based decision-making has formalized such choice through reinforcement learning models for two extreme strategies. These strategies are model-free (MF), which is an automatic, stimulus-response type of action, and model-based (MB), which bases choice on cognitive representations of the world and causal inference on environment-behavior structure. The emphasis of examining the neural substrates of value-based decision making has been on the striatum and prefrontal regions, especially with regards to the "here and now" decision-making. Yet, such a dichotomy does not embrace all the dynamic complexity involved. In addition, despite robust research on the role of the hippocampus in memory and spatial learning, its contribution to value-based decision making is just starting to be explored. This paper aims to better appreciate the role of the hippocampus in decision-making and advance the successor representation (SR) as a candidate mechanism for encoding state representations in the hippocampus, separate from reward representations. To this end, we review research that relates hippocampal sequences to SR models showing that the implementation of such sequences in reinforcement learning agents improves their performance. This also enables the agents to perform multiscale temporal processing in a biologically plausible manner. Altogether, we articulate a framework to advance current striatal and prefrontal-focused decision making to better account for multiscale mechanisms underlying various real-world time-related concepts such as the self that cumulates over a person's life course.

Keywords: decision making; hippocampus; reinforcement learning; self; successor representation.

Figure 1

(A) (Top) Illustrative example of an agent traversing between 4 states s₁ to s₄, the relationship between which is depicted by the arrows corresponding to allowed state transitions. (Bottom) The successor matrix for this state diagram, where γ depicts the discount factor. Under a random walk policy, each column of this matrix has a value of 1 on its diagonal and gradually decreasing in either direction. In terms of occupancy relative to other states, these columns resemble hippocampal place fields. (B) Comparison between different RL agents for the efficiency-flexibility trade-off. The efficiency axis represents the degree to which the model requires costly versus cheap computation. The flexibility axis represents the degree to which the agent can adapt flexibly to changes in the environment, i.e., how much new data needs to be gathered for value estimates to converge to the correct value. Figure adapted from Gershman (2018). (C) The same SR matrix as shown in panel (A), but for a task where now there is no more state transition from s₃ to s₄. This is an example of a transition revaluation. In this case, the SR can correctly update the changed transition from s₃ to s₄ but is unable to update the entries preceding this transition, i.e., the transitions from s₁ to s₄ and s₂ to s₄ should also go to 0.

Figure 2

Overview of spatial sequences in the hippocampus. (A) Hippocampal replay: (Left) Firing of hippocampal place cells as a rat runs on a maze. The cells are successively activated as the animal traverses through their place fields (color-coded on the maze), forming a sequence. (Center) Place cells indicate spatial location by firing maximally at their preferred location (known as a place field). (Right) Sharp-wave ripples (SWRs) occur during NREM sleep or quiet wakefulness and is associated with increased hippocampal population activity. During a SWR, place cell trajectories that were experienced during wakefulness are “replayed.” Adapted with permission from Zielinski et al. (2017). (B) Encoding of spatial location within theta sequences. (Left) The location of the animal is encoded via a phase code of the theta oscillation, with past locations being represented on the negative phase and future locations on the positive phase of the theta oscillation. The current location is represented at the trough of the oscillation. Reprinted from Petersen and Buzsáki (2020), with permission from Elsevier. (Right) During a deliberative decision task, hippocampal population activity nested within theta sequences sweeps forward in time, representing future spatial options. Reproduced from Redish (2016) with permission from SNCSC.