Geometry of abstract learned knowledge in the hippocampus

Abstract

Hippocampal neurons encode physical variables^1-7 such as space¹ or auditory frequency⁶ in cognitive maps⁸. In addition, functional magnetic resonance imaging studies in humans have shown that the hippocampus can also encode more abstract, learned variables^9-11. However, their integration into existing neural representations of physical variables^12,13 is unknown. Here, using two-photon calcium imaging, we show that individual neurons in the dorsal hippocampus jointly encode accumulated evidence with spatial position in mice performing a decision-making task in virtual reality^14-16. Nonlinear dimensionality reduction¹³ showed that population activity was well-described by approximately four to six latent variables, which suggests that neural activity is constrained to a low-dimensional manifold. Within this low-dimensional space, both physical and abstract variables were jointly mapped in an orderly manner, creating a geometric representation that we show is similar across mice. The existence of conjoined cognitive maps suggests that the hippocampus performs a general computation-the creation of task-specific low-dimensional manifolds that contain a geometric representation of learned knowledge.

Extended Data Figure 1.. Characterization of CA1 neural variability in the accumulating towers task.

a, Each heatmap represents one neuron and the trial-by-trial activity of that neuron in the towers task for left-choice trials. Each row in each heatmap is the ΔF/F (normalized within each session) of the neuron in that trial. b, Same as in a, but for the alternation task. Note that the single trial activity appears more variable in the towers task and more reliable in the alternation task, consistent with the results that evidence is also being represented by neurons in the towers task. c, Neural activity (ΔF/F normalized within each neuron) of cells significantly encoding evidence, sorted by activity in half the trials (left), and plotted using the same sorting in the other half of the trials (right).

Extended Data Figure 2.. Place fields in evidence-by-position (E×Y) space.

a, Each heatmap shows the mean ΔF/F of a neuron with significant mutual information in E×Y space. b, Scatterplot of the mutual information in R_E×Y space vs E×Y space for each cell with significant information in E×Y space (n=917 neurons). R_E is randomized evidence. c, Same as in b, but for E×R_Y space vs E×Y space. R_Y is randomized position. d, 29% of imaged neurons had significant mutual information in E×Y space, while 16% had significant mutual information for position only and 6% had significant mutual information for evidence only. e, Of the cells with significant mutual information in E×Y space, 89.9% had significantly more information in E×Y space than just place or evidence information alone, while 9.8% could not be differentiated from place cells, and 0.3% could not be differentiated from evidence cells (see Methods). f, The probability of a cell having significant mutual information in E×Y space is significantly greater than the joint probability of a cell being a place cell and a cell being an evidence cell (two-tailed Wilcoxon signed rank test, n=7 animals, *p=0.016; error bars: mean±SEM). g, Cells with significant mutual information in E×Y space had 1.7±0.03 SEM firing fields (n=917 cells).

Extended Data Figure 3.. Dimensionality of an earlier training stage.

During the training of the towers task, animals proceed through various stages of training. In one of these training stages, animals perform a task virtually identical to the towers task, except that visual cues only show up on one side of the maze. a, The intrinsic dimensionality of the one-side cues task is ~4.2 [4.0, 4.5] (n=4 animals, bracket values represent 95% bootstrapped confidence interval; error bars: mean±95% bootstrapped confidence intervals for each animal). b, Intrinsic dimensionality of the one-side cues task is significantly lower than the dimensionality of the towers task (two-tailed Wilcoxon rank sum test, n=7 towers task animals and n=4 one-side cues task animals, *p=0.042; error bars: mean±SEM). c, Choice-specific place cell sequences in the one-side cues task, similar to Fig. 2a. Sequences are divided into left-choice (top row), right-choice (middle row) and non- (bottom row) preferring cells. Data is split between left-choice trials (left column) and right-choice trials (right column). Cells are shown in the same order within each row group. ΔF/F was normalized within each neuron.

Extended Data Figure 4.. Cross-validation methods and results demonstrating how neural activity from single neurons is captured by coordinated population activity.

a, Illustration of the cross-validation method to calculate the decoding index in Fig. 3c. Data is split for training (solid colors) and testing (shaded colors). With the training data, a map is obtained from ΔF/F to latent dimensions and back. This map is evaluated on the test data. b, To assess the performance of the map, we concatenate the neuron x time data in the test block and reconstructed test block into two vectors and calculate the correlation coefficient from the elementwise pairwise comparison of the vectors. The correlation coefficient was averaged across 10 individually held-out trials to yield the decoding index. c, Illustration of a similar analysis where the activity of a single cell is decoded from a manifold fit to the rest of the neural population. One neuron (red) is removed before using MIND to obtain a set of latents. Next, in the training data (solid green), a map is calculated from the manifold to the held-out neuron’s activity. The map is then used to predict the test data (shaded green). The correlation coefficient is calculated as in b and averaged over 5-folds as the decoding index. d, Example of neural activity from 40 individually reconstructed neurons, where activity of each neuron was decoded from the 5-dimensional manifold fit to the other cells following procedures in c (comparable to Fig. 3b, where the method in panels a and b was used). ΔF/F is normalized to the maximum ΔF/F in the window shown. e, Cross-validated correlation coefficients between activity of individual neurons in the real and reconstructed data, where reconstruction was accomplished with d-dimensional embeddings of the neural manifold. Decoding index is the correlation coefficient between the predicted and real ΔF/F of the held-out ROIs (n=7 animals; error bars: mean±SEM).

Extended Data Figure 5.. Task manifold and neural manifold encode different variables.

a, The visual space of the accumulating towers task across a representative session. Shown is the mean luminance of the virtual reality visual field as a function of position in the T-maze. Four example frames are shown below. Note the high variability of luminance during the cue period, where bright towers are randomly presented on the left and right walls. b, Performing dimensionality reduction on the pixels’ time series in the raw video stream using MIND reveals a low-dimensional manifold, reflecting the visual sensory structure of the accumulating towers task. Plotting luminance (top) and evidence (bottom) on the manifold reveals that luminance is represented as a smooth gradient, whereas evidence requires memory and is thus absent on the task manifold. c, same as in b, but showing the neural manifold obtained from the animal that ran this session (Fig. 3f). Notice the absence of a luminance representation, but the emergence of evidence.

Extended Data Figure 6.. Decoding other variables from the neural manifold.

a, Similar to Fig. 3f, view angle is plotted as color on the 3-dimensional embedding of the manifold. b, The 5 latent variables of the neural manifold embedded in 5-dimensional space are better predicted by gaussian process regression from view angle and evidence values than from view angle and shuffled evidence values (two-tailed Wilcoxon signed rank test, n=7 animals, *p=0.016; error bars: mean±SEM). Decoding index is the correlation coefficient between the predicted manifold values and true manifold values, averaged over the 5 dimensions of the manifold. c, Same as in b, but for decoding manifold values using position and velocity. The addition of velocity to position information significantly improves the decoding of manifold values (two-tailed Wilcoxon signed rank test, n=7 animals, *p=0.016; error bars: mean±SEM). d, Same as in b, but for decoding using position and time. The addition of time information does not significantly increase how well manifold values are decoded (two-tailed Wilcoxon signed rank test, n=7 animals, ^nsp=0.30; error bars: mean±SEM). e, We used PCA to separate the correlated and orthogonal dimensions between evidence and view angle and decoded both PC1 (correlated) and PC2 (orthogonal) from the neural manifold embedded in 5-dimensional space (n=7 animals; error bars: mean±SEM). Decoding index is the correlation coefficient between the predicted PC and true PC values. f, View angle is better decoded from the neural manifold (5-dimensional embedding) in the towers task (“Tow”), when evidence is also present, than in the alternation task (“Alt”) when evidence is not present (two-tailed Wilcoxon rank sum test, n=7 towers task animals and n=7 alternation task animals, p=0.07; error bars: mean±SEM). Decoding index is the correlation coefficient between the predicted and true view angle values. g, Average view angle trajectories, separated between left- and right-choice trials, for the towers task (n=7; blue/thin) and the alternation task (n=7; red/thin) animals. Thick lines represent averages across animals. h, Average view angle values in the towers task (n=7; blue/thin) and the alternation task (n=7; red/thin) over all trials. Thick lines and shaded area: mean±95% bootstrapped confidence interval. i, Accuracy in predicting the upcoming choice (left), the animal’s choice in the previous trial (center), and whether the previous trial was rewarded (right) from d-dimensional embeddings of the neural manifold (n=7 animals; error bars: mean±SEM).

Extended Data Figure 7.. Examples of sequences in CA1 neural activity.

a, Schematic to describe how “doublets” were defined. Orange and green are calcium traces of the cells under consideration. Grey is the calcium trace of a third cell. b, 25 examples of doublets in a single session from one animal. Each panel shows traces for trials in which the doublet was present. Orange traces are the neural activity from the first cell in the doublet, while green traces are the neural activity from the second cell in the doublet. Heatmaps represent the normalized neural activity of each cell across all trials in the session.

Extended Data Figure 8.. Neural activity generated by trajectories through the task.

a, Trajectories through evidence and position in one session of the task. Each thin line represents a fit with a cubic spline to a single trial, while thick lines represent fits over all trials in which the animal was supposed to turn left or right. b, Shown is the average change of position and evidence over time across trials in a single session for a set of representative states in evidence and position space. c, Conceptual diagram showing four trajectories through the neural manifold in right choice trials. Two different doublets are activated because the trajectories pass through their firing fields. d, Shuffling trial IDs within right choice trials will disrupt doublet activity while maintaining trial-averaged place and choice preferences of each cell.

Extended Data Figure 9.. Choice-predictive sequences in CA1 neural activity.

a, Distribution of the values in Fig. 4b. b, Distribution of the values in Fig. 4c. c, Distribution of the values in Fig. 4f. d, Receiver operating characteristic (ROC) curves for sequential activity predicted from the 5-dimensional embedding of the manifold compared to sequential activity in real data (n=7 animals). e, Similar to a, but for triplets. Inset shows that triplets are significantly more likely to appear in the real data than in the shuffled dataset where trial IDs were shuffled (two-tailed paired t-test, n=34737 triplets, ****p<0.0001). f, Similar to c, but for triplets, showing that left- and right-choice predictive triplets from real data are more predictive than triplets obtained from the shuffled dataset where trial IDs were shuffled (left inset: left-predictive, two-tailed paired t-test, n=1135 triplets, real vs shuffle: ****p<0.0001; right inset: right-predictive, two-tailed paired t-test, n=1755 triplets, real vs shuffle: ****p<0.0001). g, Left-choice predictive triplets are significantly more predictive than instances where the first two cells in the triplet fire, but the third does not, or when the third cell fires alone (two-tailed paired t-tests, Bonferroni corrected, n=1135 triplets, 1→2→3 vs 1→2→not 3: ****p<0.0001; 1→2→3 vs not 1→not 2→3: ****p<0.0001; 1→2→not 3 vs not 1→not 2→3: ^nsp=0.78). h, Importantly, for left-choice predictive triplets, in trials where cells 1 and 2 fire, but cell 3 does not, significantly more trials end with the animal turning right than the same instances in the shuffled dataset (right panel: two-tailed paired t-test, n=1135 triplets, real vs shuffle: ****p<0.0001). i, Same as g, but for right-choice predictive triplets (two-tailed paired t-tests, Bonferroni corrected, n=1755 triplets, 1→2→3 vs 1→2→not 3: ****p<0.0001; 1→2→3 vs not 1→not 2→3: ****p<0.0001; 1→2→not 3 vs not 1→not 2→3: ^nsp=1.0). j, Same as in h, but for right-choice predictive triplets (right panel: two-tailed paired t-test, n=1755 triplets, real vs shuffle: ****p<0.0001). For boxplots, boundaries: 25th/75th percentiles, midline: median, whiskers: min/max.

Figure 1.. Imaging CA1 neural activity in mice performing the accumulating towers task.

a, Schematic of the task in which head-fixed mice navigate in a virtual reality evidence accumulation T-maze task. Insets show example views from the animals’ perspective (top). While animals (n=7) perform the task, 2-photon calcium imaging records hippocampal CA1 neural activity (bottom). b, Psychometric curves of mice performing the towers task (grey lines: n=7 animals, black line: metamouse combining data across animals; error bars: mean±binomial confidence interval). c, Logistic regression showing that mice use evidence throughout the cue period (grey lines: n=7 animals, black line: metamouse combining data across animals; error bars: mean±SEM). d, Firing fields of right-choice selective place cells would not depend on evidence and would thus divide a joint Evidence-by-Position (E×Y) space into two halves (top). Two right choice trials would generate the same neural sequence (bottom). e, Alternatively, if hippocampal neurons encoded evidence jointly with position, smaller firing fields dividing up evidence would appear in E×Y space (top), and two right choice sequences could have different neural sequences depending on the evidence values traversed (bottom).

Figure 2.. CA1 neurons jointly encode position and evidence in an evidence accumulation task.

a, Choice-specific place cell sequences, divided into left-choice (top row), right-choice (middle row) and non- (bottom row) preferring cells. Cells are shown in the same order within each row group. ΔF/F was normalized within each neuron. b, CA1 neurons have firing fields in accumulated evidence space (# right towers - # left towers). c, Example of the average neural activity of a single neuron in joint evidence-by-position (E×Y) space. d, Twenty-five neurons with significant information in E×Y space. Each color represents one cell, and surfaces represent neural activity that exceeds 2 standard deviations (SD) above the shuffled means (Extended Data Fig. 2a). e, Mutual information of cells found to have significant information in E×Y space is significantly greater than mutual information in 2D spaces where either evidence (R_E) or position (R_Y) has been randomized (two-tailed paired t-tests, Bonferroni corrected, n=917 neurons, E×Y vs R_E×Y: ****p<0.0001; E×Y vs E×R_Y: ****p<0.0001; R_E×Y vs E×R_Y: ****p<0.0001). For boxplots, boundaries: 25th/75th percentiles, midline: median, whiskers: min/max.

Figure 3.. Geometric representation of task variables on low-dimensional neural manifolds.

a, The average cumulative number of neighboring neural states as a function of the geodesic distance, plotted on a log-log axis, revealing a ~5.4 [4.8, 6.0] dimensional manifold (n=7 animals, bracket values represent 95% bootstrapped confidence interval; error bars: mean±95% bootstrapped confidence intervals for each animal). b, Example of neural activity from 40 neurons (left) and the activity of those same 40 neurons reconstructed from the five latent variables obtained from embedding the manifold in a 5-dimensional Euclidean space (right). ΔF/F is normalized to the maximum ΔF/F in the window shown. c, Reconstruction of held-out neural data from d-dimensional embeddings of the neural manifold. Decoding index is the correlation coefficient between the predicted and real ΔF/F data in held-out trials. d, Each point in this plot is a location in the 3-dimensional embedding of the manifold at one time point in an imaging session. Colored points represent ΔF/F values that are 3 standard deviations above the mean activity for one example cell. e, Firing field of five cells, each in a different color, plotted on the manifold. f, Position (left) and evidence (right) plotted as color on the 3-dimensional embedding of the manifold. Black arrows represent two hypothetical trajectories through manifold space that would traverse through position space and increasing left or right evidence values. g, Decoding position (left) and evidence (right) from d-dimensional embeddings of the manifold using gaussian process regression (GPR). Decoding index is the correlation coefficient between the predicted and true position or evidence values. The shaded area and line represent the mean decoding index ±SEM using GPR on the top 10% of neurons with the highest mutual information for position or evidence as inputs. h, Schematic of the hyperalignment method for aligning two manifolds (see Methods). i, Decoding index of position and evidence for the hyperalignment, i.e. the best decoding that can be done using one of the six other manifolds, vs. decoding with GPR in the same animal for the 5-dimensional embedding of the manifolds (two-tailed Wilcoxon signed rank test, n=7 animals, position: *p=0.016; evidence: ^nsp=0.81). j, Percent of geometry shared across animals. The majority of manifold geometry (n=7 animals, position: 69%±9% SEM; evidence: 75%±10% SEM) is shared between the best pairs of animals. In panels c, g, i, and j, error bars: mean±SEM (n=7 animals).

Figure 4.. Sequential activity of CA1 neurons in single trials is predictive of behavior and explained by the manifold.

a, Two examples of doublets, where two neurons consistently fire one after the other. In example 1, activity does not appear to be tied to time (left) or position (right). Highlighted trials in cyan and purple are the same trials plotted in d. In example 2, activity in both neurons appears to be related to time/position in the trial. b, Doublets appear more frequently in real data than in a shuffled dataset (two-tailed paired t-test, n=16088 doublets, real vs shuffle: ****p<0.0001). c, Doublets are asymmetric (two-tailed paired t-test, n=16088 doublets, real vs shuffle: ****p<0.0001). Directionality index is defined as the number of times cell 1 fires before cell 2 in a trial minus the number of times cell 2 fires before cell 1 in a trial. d, Example showing how events from cell 1 (orange) and cell 2 (green) of a doublet are separated in manifold space. Cyan and purple lines each represent a trial trajectory between when cell 1 and cell 2 fire. e, Amount of time between when cell 1 and cell 2 fire plotted against distance in manifold space. f, Left- and right-choice predictive doublets (left and right panels, respectively) are significantly more predictive of upcoming choice than doublets generated from shuffled data where trial IDs were shuffled (left-predictive, two-tailed paired t-test, n=922 doublets, real vs shuffle: ****p<0.0001; right-predictive, two-tailed paired t-test, n=1227 doublets, real vs shuffle: ****p<0.0001). For boxplots, boundaries: 25th/75th percentiles, midline: median, whiskers: min/max.