Reinstatement and transformation of memory traces for recognition

Abstract

Episodic memory relies on the formation and retrieval of content-specific memory traces. In addition to their veridical reactivation, previous studies have indicated that traces may undergo substantial transformations. However, the exact time course and regional distribution of reinstatement and transformation during recognition memory have remained unclear. We applied representational similarity analysis to human intracranial electroencephalography to track the spatiotemporal dynamics underlying the reinstatement and transformation of memory traces. Specifically, we examined how reinstatement and transformation of item-specific representations across occipital, ventral visual, and lateral parietal cortices contribute to successful memory formation and recognition. Our findings suggest that reinstatement in temporal cortex and transformation in parietal cortex coexist and provide complementary strategies for recognition. Further, we find that generalization and differentiation of neural representations contribute to memory and probe memory-specific correspondence with deep neural network (DNN) model features. Our results suggest that memory formation is particularly supported by generalized and mnemonic representational formats beyond the visual features of a DNN.

Fig. 1.. Task, sample, behavioral performance, and electrode coverage.

(A) Behavioral paradigm. Participants completed a visual scene memory recognition task with distinct encoding and retrieval blocks. (B) Sample distribution of age (15.61 ± 5.92 years; left) and recognition memory performance Pr (Hit – false alarm; 0.41 ± 0.19; right). (C) Corrected recognition memory performance (Hits – false alarms) across development. (D) Regions of interest (ROIs) based on Brodmann areas (BAs): Occ (BA17-19); pTC (BA37); aTC (BA20); LPC (BA39 + 40). (E) Heatmap of contributing electrodes in these ROIs. Coordinate axes labels: A, anterior; S, superior; L, lateral; P, posterior; V, ventral; ECoG, electrocorticography.

Fig. 2.. Representational similarity analyses.

(A) RSA. We computed pairwise similarities between different scenes within the same experimental phase (EES; RRS), and between the same and different scenes across experimental phases (ERS). (B) Time-frequency features used for RSA. We extracted item-specific distributions of power values across frequencies and electrodes within a given ROI in various time windows. (C) Schematic ERS matrix. Similarities during encoding and retrieval of the same scenes (ERS_Same) are depicted on the diagonal (blue) of this matrix, while similarities between different scenes (ERS_Diff) are depicted on the off-diagonal (orange). Item-specific ERS scores were computed by subtracting the average off-diagonal ERS score per row from its on-diagonal value: ERS_Item = ERS_Same – avg. (ERS_Diff). We compared ERS_Item values between remembered and forgotten trials, testing for a memory benefit of either reinstatement (remembered > forgotten) or transformation (forgotten > remembered). (D) Schematic pairwise similarity matrix during encoding (EES) or retrieval (RRS) reflecting pairwise representational similarities between different items. During both encoding and retrieval, we compared between-item similarities of remembered versus forgotten scenes. We compared the pairwise similarities between remembered and forgotten scenes, testing for memory benefits of generalization (remembered > forgotten) or differentiation (forgotten > remembered).

Fig. 3.. Reinstatement of item-specific representations in anterior temporal cortex.

(A) Contributing electrodes in the aTC region of interest (ROI) (34 participants, 107 channels). (B) Left, higher ERS_Item scores for remembered versus forgotten scenes. Color reflects T values across subjects for each time point. Black contour indicates cluster of time points during which ERS_Item values were significantly larger for remembered versus forgotten scenes after cluster-based correction for multiple comparisons across time points. Right, illustrative plot of ERS_Item similarities (Spearman’s ρ) per subject for remembered (amaranth) versus forgotten (blue) scenes, averaged across time points of significant cluster. (C) ERS_Same (blue, diagonal) and ERS_Diff (orange, off-diagonal) values of remembered and forgotten items showing a memory effect of ERS_Same but not ERS_Diff scores. (D) No relationship between reinstatement effects on memory with either age (left) or performance (right). Circled * indicates P < 0.05 for one-sample T test. Line * indicates P < 0.05 for paired-sample T test.

Fig. 4.. Transformation of item-specific representations in lateral parietal cortex (LPC).

(A) Contributing electrodes in LPC region of interest (ROI) (38 participants, 466 channels). D, dorsal. (B) Left, lower ERS_Item scores for remembered versus forgotten scenes. Color reflects T values across subjects for each time point. Black contour indicates cluster of time points during which ERS_Item values were significantly lower for remembered versus forgotten scenes after cluster-based correction for multiple comparisons across time points. Right, illustrative plot of ERS_Item similarities (Spearman’s ρ) per subject for remembered (red) versus forgotten (blue) scenes, averaged across time points of significant cluster. (C) ERS_Same (blue, diagonal) and ERS_Diff (orange, off-diagonal) values of remembered and forgotten items showing a memory effect of ERS_Same but not ERS_Diff scores. (D) No significant relationship between transformation benefit for memory and participants’ age. (E) Left, significant relationship between transformation benefit for memory and participants’ performance. Colors indicate grouping following median split. Right, transformation effects in low- and high-performing participants (median split). (F) Relationship between ERS_Item magnitude in aTC and LPC: The magnitude of ERS_Item values in aTC is associated with ERS_Item magnitude in LPC. Circled * indicates P < 0.05 for one-sample T test. Line * indicates P < 0.05 for paired-sample T test. * indicates uncorrected P < 0.05 for main effect of LPC ERS_Item predicting aTC ERS_Item.

Fig. 5.. Generalization and differentiation of memory representations.

Memory effects of generalization (i.e., EES/RRS remembered > forgotten) or differentiation (i.e., EES/RRS remembered < forgotten) during encoding (EES, top) and retrieval (RRS, bottom). Colored horizontal lines indicate time points of significant memory effects after cluster-based correction for multiple comparisons across time points, separately for each region of interest (ROI).

Fig. 6.. Deep neural networks (DNNs) reveal memory-relevant representational formats.

(A) Top, schematic of cDNN architecture trained to classify scenes (PlacesNet). Bottom, multidimensional scaling (MDS) plots for scene stimuli used in the experiment showing more pronounced categorical (indoor/outdoor) clustering of representations in higher versus lower DNN layers. (B) RSA comparing neural representational similarity matrices (RSMs) during encoding (EES) and retrieval (RRS) with RSMs from the individual DNN layers. During both encoding and retrieval, we correlated trial-wise RSMs observed in neural data with RSMs in each DNN layer and then conducted a follow-up test for differences of neural-DNN similarities depending on memory (i.e., remembered versus forgotten trials). (C) Time-resolved analysis of encoding-DNN correspondence for remembered versus forgotten items (left) and separately for only remembered (middle) or forgotten (right) items. Dashed vertical lines indicate on- and offset of encoding and retrieval time periods relevant for reinstatement in aTC. Horizontal colored bars indicate time points of cluster-corrected effects across time points, separately for each layer of the DNN.