Neural signatures of reinforcement learning correlate with strategy adoption during spatial navigation

Abstract

Human navigation is generally believed to rely on two types of strategy adoption, route-based and map-based strategies. Both types of navigation require making spatial decisions along the traversed way although formal computational and neural links between navigational strategies and mechanisms of value-based decision making have so far been underexplored in humans. Here we employed functional magnetic resonance imaging (fMRI) while subjects located different objects in a virtual environment. We then modelled their paths using reinforcement learning (RL) algorithms, which successfully explained decision behavior and its neural correlates. Our results show that subjects used a mixture of route and map-based navigation and their paths could be well explained by the model-free and model-based RL algorithms. Furthermore, the value signals of model-free choices during route-based navigation modulated the BOLD signals in the ventro-medial prefrontal cortex (vmPFC), whereas the BOLD signals in parahippocampal and hippocampal regions pertained to model-based value signals during map-based navigation. Our findings suggest that the brain might share computational mechanisms and neural substrates for navigation and value-based decisions such that model-free choice guides route-based navigation and model-based choice directs map-based navigation. These findings open new avenues for computational modelling of wayfinding by directing attention to value-based decision, differing from common direction and distances approaches.

Figure 1

Wayfinding task and behavioral results. (A) Layout of the grid-world. The Virtual Reality (VR) environment consisted of a 5 by 5 grid of rooms. Each square represents a room which contained distinct furniture and objects to distinguish individual rooms. Black square represents starting position, colored squares reward locations and the number represents the order in which they need to be found. The wayfinding task consisted of three phases: encoding, retrieval, and search phase. During the search phase, subject had to locate one randomly chosen reward at each trial, each time starting from a different starting position. (B) Screenshots of the virtual reality environment. Each room is furnished with distinct objects to allow subjects to distinguish and recognize individual rooms. At each room (decision point) subjects could choose up to three directions (corner rooms had either one or two directions to choose). After a choice was made, an animation was leading to the room in the selected direction; this movement lasted 2.5–3 seconds jittered uniformly. The next room and, if applicable, the reward were presented. (C) Path from a representative participant (subject no. 1) who exhibited a tendency towards route-based strategy. During the encoding phase, the subject established by repetition a fixed route from one reward to the other. During the search phase, the subject still followed the established route to reach the reward when it started from a location on that previous route. However, when the subject started from a position which was not part of the original route, she could locate the reward room using the shortest possible path.

Figure 2

Reinforcement learning models and model fits. The top panel displays action values, showing how valuable it is to move along the route in a certain state; the bottom panel shows the probability of taking certain actions in those state based on the action values. Black numbers are state values, blue numbers are probabilities of chosen action, green values refer to probabilities of other not chosen actions. Note that not all probabilities for non-preferred actions are shown. (A) Model free valuation based on the SARSA (λ) algorithm. After reaching a reward this algorithm updates the values only along the traversed path. (B) Model-based valuations derived from dynamic programming. The model-based algorithm updates values not only along the taken path, but across the entire grid world.

Figure 3

Correlation between navigation indices and ω parameters (n = 27). (A) Significant positive correlation between I_PATH and ω parameters. (B) Significant positive correlation between I_STEP and ω parameters. (C) Significant negative correlation between I_ROUTE and ω parameters for the fMRI experiment. I_PATH, I_STEP, I_ROUTE, and ω are averaged values for individual subject across three different phases of the wayfinding task.

Figure 4

Correlations of model predicted values with BOLD signals. (A) Correlates of model-free valuations in medial/vmPFC, striatum, and retrosplenial cortex. (B) Correlates of model-based valuation in parahippocampal and medial temporal lobe region as well as the left retrosplenial cortex. Displayed results are significant at P < 0.05 whole brain FWE corrected at the cluster level.

Figure 5

Correlation between subjects’ relative degree of model-based behavior (ω) and β-parameter estimates of the model-free parametric regressor. (A) In vmPFC, we found a significant negative correlation between the ω parameter in the behavioral hybrid model and β-estimates in the GLM from the parametric regressor of model-free values. That means, across subjects, the larger the relative degree of model-free choice behavior of a subject, the stronger was her representation of model-free values in the BOLD signal in vmPFC. No such relationship was found for the model-based value regressor. (B) In the parahipocampal gyrus, we found a significant positive correlation between ω and β- estimates from the parametric regressor of model-free values. The larger the relative degree of model-based choice behavior in a subject, the bigger was her representation of model-based values in the BOLD signal in the parahipocampal gyrus. No such correlation was found for the model-free value regressor.