Differential encoding of temporal context and expectation under representational drift across hierarchically connected areas

Abstract

The classic view that neural populations in sensory cortices preferentially encode responses to incoming stimuli has been strongly challenged by recent experimental studies. Despite the fact that a large fraction of variance of visual responses in rodents can be attributed to behavioral state and movements, trial-history, and salience, the effects of contextual modulations and expectations on sensory-evoked responses in visual and association areas remain elusive. Here, we present a comprehensive experimental and theoretical study showing that hierarchically connected visual and association areas differentially encode the temporal context and expectation of naturalistic visual stimuli, consistent with the theory of hierarchical predictive coding. We measured neural responses to expected and unexpected sequences of natural scenes in the primary visual cortex (V1), the posterior medial higher order visual area (PM), and retrosplenial cortex (RSP) using 2-photon imaging in behaving mice collected through the Allen Institute Mindscope's OpenScope program. We found that information about image identity in neural population activity depended on the temporal context of transitions preceding each scene, and decreased along the hierarchy. Furthermore, our analyses revealed that the conjunctive encoding of temporal context and image identity was modulated by expectations of sequential events. In V1 and PM, we found enhanced and specific responses to unexpected oddball images, signaling stimulus-specific expectation violation. In contrast, in RSP the population response to oddball presentation recapitulated the missing expected image rather than the oddball image. These differential responses along the hierarchy are consistent with classic theories of hierarchical predictive coding whereby higher areas encode predictions and lower areas encode deviations from expectation. We further found evidence for drift in visual responses on the timescale of minutes. Although activity drift was present in all areas, population responses in V1 and PM, but not in RSP, maintained stable encoding of visual information and representational geometry. Instead we found that RSP drift was independent of stimulus information, suggesting a role in generating an internal model of the environment in the temporal domain. Overall, our results establish temporal context and expectation as substantial encoding dimensions in the visual cortex subject to fast representational drift and suggest that hierarchically connected areas instantiate a predictive coding mechanism.

Figure 1. Presenting images in a variety of stimulus blocks determines single cell activity along the visual hierarchy.

a) Experimental pipeline from developing transgenic mice, to recording population responses in the visual cortex, to finally an open dataset available to the public in the DANDI archive.b) Mice passively view natural images in different stimulus blocks while neural activity is recorded from V1, PM, or RSP. Images are either presented in random order (yellow and green) or in common, expected sequences with the occasional oddball interleaved (red), or in transition control (blue) which only preserves pairwise transitions. c) Single cell PSTHs to natural images in each stimulus blocks. Each row shows a representative average response from each recorded area, with the shaded area showing the variance across trials. i-ii: Superficial V1 neurons. iii- iv: Deep V1 neurons. v: Superficial PM neuron. vi: Deep PM neuron. vii: Superficial RSP neuron. viii: Deep RSP neuron. d) One-way ANCOVA results, accounting for the locomotion of the animal. Left: A small fraction of cells in V1, PM, and RSP are selective to expected images in all stimulus blocks. Right: A larger fraction of cells in V1 and PM are selective to unexpected images in the sequence and transition control blocks as compared to the randomized control ones. Error bars indicate standard deviation across depths. e) Using responses to expected images (ABCD) and unexpected images (X), we can assess whether a cell is selective to the stimulus blocks in which the image was presented. Error bars indicate standard deviation across different images. f) Histogram of Oddball Response Indices calculated for each cell in the 3 areas.

Figure 2.. Decoding of natural images across different stimulus blocks.

a) Schematic showing linearly separable population responses to natural images presented in the main-sequence. b & c) Example confusion matrices show significant decoding of main-sequence images using superficial V1 or RSP population responses during the sequence stimulus block. d) Significant decoding, denoted by the gray asterisk (t-test compared to shuffle distribution, p < 0.05), of expected images along the visual hierarchy in different stimulus blocks. All depths of V1 and superficial RSP, show an increase in decoding performance for the sequence and transition control blocks relative to randomized control (black asterisk, Wilcoxon signed-rank test, p < 0.05). In PM, only the transition control block was significantly different compared to randomized control. Deep RSP remains at chance level for all stimulus blocks. Randomized control pre not shown for clarity. e) Schematic showing linearly separable population responses to unexpected natural images. f & g) Example confusion matrices show significant decoding of unexpected oddball images using V1 population responses during the sequence stimulus block, but not for RSP population responses. h) V1 and PM show significant decoding of unexpected natural images in all stimulus blocks (gray asterisk). V1 shows an increase of decoding performance when unexpected images disrupt 2-image sequences and even more so to 4 image sequences, as compared to randomized control. In PM, decoding performance of unexpected images is larger for the sequence block compared to transition control and randomized control, which are statistically insignificant from each other. Superficial RSP shows slight decoding performance to oddball images in the sequence block, but no others. Deep RSP remains at chance level.

Figure 3.. Decoding responses to expected and unexpected natural images reveals possible predictive coding mechanism in RSP

a) Schematic showing the task of classifier. b) PCA of V1 PSTHs for the sequences immediately preceding and following an unexpected event (X_i). Oddball representation in PCA is distinct from expected image D. c) PCA of superficial RSP PSTHs for the sequences immediately preceding and following an unexpected event (X). Oddball representation in PCA space overlaps that of the expected image D. d) Significant decoding of main-sequence and oddball images along the visual hierarchy in the sequence. Hash pattern for RSP denotes deep layers (> 350 μm) e) Example confusion matrices from superficial V1, superficial PM, and superficial/deep RSP. f) Histograms per area of the relative miss rates of main sequence images (A instead of B, B instead of C, etc) compared to that misclassifying images D and X. Superficial RSP reveals a significant increase in the DX miss-rate relative to the ABCD miss-rate.

Figure 4.. Generalization performance under representational drift.

a) Schematic showing linearly separable population responses to the same natural image presented in different stimulus blocks. Decoding the time or stimulus block in which the same image is presented is our measure of representational drift. b) Example confusion matrices showing significant decoding of stimulus block using population responses from V1, PM, and superficial RSP. Representations from Deep RSP are unable to perform the decoding task. Epochs include the four main stimulus blocks, plus 2 additional epochs from early and late in the sequence block. c) Schematic showing population responses to main sequence images in different epochs of the session, with training and testing occurring using data from different stimulus blocks (epochs). d) Matrices show the generalization performance between epochs of the same sessions as panel b. Each element in the matrix represents a diagonal of a confusion matrix. In V1, representations of main sequence images generalize across stimulus blocks, while in PM, this phenomenon is less persistent. For superfical and deep RSP, generalization does not manifest. e) Example representational similarity matrices for V1, PM, and RSP show the correlation structure within and between stimulus blocks. Trials of main-sequence images are reordered to show correlations between images. V1 & PM reveal a strong correlational structure of within-epoch population responses, with comparable correlations for between-epoch blocks. Representational similarity from superficial RSP reveals a strong correlation of within-epoch population responses, with little to no correlation between-epoch blocks. Deep RSP shows little correlation between trials. All example confusion matrices, generalization matrices, and representational similarity matrices are of the same session per area.