Memorable Audiovisual Narratives Synchronize Sensory and Supramodal Neural Responses

Abstract

Our brains integrate information across sensory modalities to generate perceptual experiences and form memories. However, it is difficult to determine the conditions under which multisensory stimulation will benefit or hinder the retrieval of everyday experiences. We hypothesized that the determining factor is the reliability of information processing during stimulus presentation, which can be measured through intersubject correlation of stimulus-evoked activity. We therefore presented biographical auditory narratives and visual animations to 72 human subjects visually, auditorily, or combined, while neural activity was recorded using electroencephalography. Memory for the narrated information, contained in the auditory stream, was tested 3 weeks later. While the visual stimulus alone led to no meaningful retrieval, this related stimulus improved memory when it was combined with the story, even when it was temporally incongruent with the audio. Further, individuals with better subsequent memory elicited neural responses during encoding that were more correlated with their peers. Surprisingly, portions of this predictive synchronized activity were present regardless of the sensory modality of the stimulus. These data suggest that the strength of sensory and supramodal activity is predictive of memory performance after 3 weeks, and that neural synchrony may explain the mnemonic benefit of the functionally uninformative visual context observed for these real-world stimuli.

Keywords: episodic encoding; intersubject correlation; multisensory integration; naturalistic stimuli.

Figure 1.

Illustration of the behavioral task. Subjects were exposed to one of five conditions: A Only, AV, AVsc, V Only, or No Stim (not shown). The sound clip represented by the waveform is “…and he would buy me a hotdog the size of my head…” Three weeks after stimulus presentation, and without prior warning, subjects were asked to complete an on-line questionnaire with 72 four-alternative forced-choice questions. The question asked about this segment of the stimulus was “What would Rocco do with the narrator when they went for walks?” Answer options were as follows: “a. Buy him a hot dog; b. Buy him a milkshake; c. Buy him candy; d. Tell him stories.” Still images from “Sundays at Rocco’s,” a StoryCorps animated short produced by Lizzie Jacobs and Mike Rauch, reproduced here with permission from StoryCorps.

Figure 2.

A, B, Memory performance for different stimulus modalities (A) and different narratives (B). A, Note that exposure to the visual stimuli (V Only, yellow) yields performance no better than chance performance (No Stim, gray). In addition to mean and SE (represented by the black horizontal and vertical lines, respectively), we also present the histogram of the distribution of accuracy values. B, For each narrative (for titles see Materials and Methods), performance is shown for A Only (purple), AV (blue), AVsc (green), and chance (No Stim, gray). Error bars represent the SEM across questions (N = 72 in A; N = 5–9 in B). *p < 0.05, **p < 0.01, ***p < 0.001.

Figure 3.

ISC A Only (purple), AV (blue), AVsc (green), and visual (V Only, yellow) stimuli. The full distribution of the ISC values are indicated by the width of the histogram bars for each condition, and gray indicates the distribution of the chance level of correlation for each modality. ISC is calculated using the sum of the three largest correlated components elicited by the presentation of the narrative (Eq. 1). Error bars (vertical lines) represent the SEM across subjects. *p < 0.05, **p < 0.01, ***p < 0.001. B, C, The multisensory boost in memory and ISC occurs for all 10 narratives. The different presentation conditions for each narrative, corresponding to separate groups of subjects, are connected with a line, and SEs across subjects are represented as horizontal and vertical bars for ISC and Memory %, respectively.

Figure 4.

Relationship between neural ISC and memory performance. A, Memory accuracy for auditory information increases with ISC in all conditions in which the auditory narrations were heard (A Only, AV, and AVsc), but not when it was missing (V Only). Each point indicates an individual subject’s ISC (Eq. 2) and memory. B, Same as A, but here, to control for the modality effect, mean values across subjects were subtracted from ISC and memory performance for each subject in that stimulus condition. Only conditions with performance significantly above chance are used.

Figure 5.

The forward model for the three most correlated components of neural activity. Each column represents the forward model (correlation between surface electrodes and component activity) obtained using either all stimuli together (combining responses across all subjects, left) or different stimulus presentations (A Only, middle-left; AV, middle-right; V Only, right). Each row represents a different component in descending order from most correlated (top) to least correlated (bottom; C1–C3). Color indicates the correlation between each scalp electrode and the component.