Representation of Real-World Event Schemas during Narrative Perception

Abstract

Understanding movies and stories requires maintaining a high-level situation model that abstracts away from perceptual details to describe the location, characters, actions, and causal relationships of the currently unfolding event. These models are built not only from information present in the current narrative, but also from prior knowledge about schematic event scripts, which describe typical event sequences encountered throughout a lifetime. We analyzed fMRI data from 44 human subjects (male and female) presented with 16 three-minute stories, consisting of four schematic events drawn from two different scripts (eating at a restaurant or going through the airport). Aside from this shared script structure, the stories varied widely in terms of their characters and storylines, and were presented in two highly dissimilar formats (audiovisual clips or spoken narration). One group was presented with the stories in an intact temporal sequence, while a separate control group was presented with the same events in scrambled order. Regions including the posterior medial cortex, medial prefrontal cortex (mPFC), and superior frontal gyrus exhibited schematic event patterns that generalized across stories, subjects, and modalities. Patterns in mPFC were also sensitive to overall script structure, with temporally scrambled events evoking weaker schematic representations. Using a Hidden Markov Model, patterns in these regions predicted the script (restaurant vs airport) of unlabeled data with high accuracy and were used to temporally align multiple stories with a shared script. These results extend work on the perception of controlled, artificial schemas in human and animal experiments to naturalistic perception of complex narratives.SIGNIFICANCE STATEMENT In almost all situations we encounter in our daily lives, we are able to draw on our schematic knowledge about what typically happens in the world to better perceive and mentally represent our ongoing experiences. In contrast to previous studies that investigated schematic cognition using simple, artificial associations, we measured brain activity from subjects watching movies and listening to stories depicting restaurant or airport experiences. Our results reveal a network of brain regions that is sensitive to the shared temporal structure of these naturalistic situations. These regions abstract away from the particular details of each story, activating a representation of the general type of situation being perceived.

Keywords: event; fMRI; narrative; perception; schema; script.

Figure 1.

Experimental stimuli. Subjects were presented with 16 narrative stimuli: half audiovisual movies (clips from movies) and half audio narratives (based on movie scripts, all read by the same voice actor). The narratives varied widely but conformed to one of two schemas: eating at a restaurant (entering the restaurant, being seated at a table, ordering food from a menu, and eating the food when it arrives) or going on a flight (entering the airport, going through security, waiting at the boarding gate, and taking a seat on the plane). The timing of these schematic events differed across stories, but all stories were ∼3 min long.

Figure 2.

Event pattern correlations between stories. a, For each pair of stories, we compute the mean pattern correlation (in a ROI) between the 4 corresponding events. We then binned these correlations depending on whether the two stories were drawn from the same schema or different schemas. b, We found that, throughout the default mode network, stories from the same schema showed significantly higher event similarity compared with stories from different schemas. This result was not driven solely by modality-specific stimulus features because it appears even when considering only pairs of stories from different modalities (audiovisual movie and audio narrative). Auditory regions were not strongly related to high-level schemas, despite the presence of audio information in all stories. Surface map thresholded at FDR < 0.05. *p < 0.05; **p < 0.01; by permutation test.

Figure 3.

Dimensionality of event patterns. To estimate the underlying dimensionality of the spatial event patterns, the SRM was used to project the group functional data into lower-dimensional spaces. With only two dimensions (the smallest possible dimensionality for which correlations can be computed), no regions show a schema effect significant at p < 0.01 and only SFG shows an effect significant at p < 0.05, indicating that a simple low-dimensional signal (such as global attentional modulation) is not sufficient to explain the results reported in Figure 2 (which used an SRM dimension of 100). Because the effects for all regions asymptote at ∼10 dimensions, we estimate that, across subjects, the fMRI signals of schematic event patterns span an ∼10-dimensional space.

Figure 4.

Visualization of schematic patterns. The spatial pattern associated with each of the schematic events was computed for two independent groups of subjects. Visualizing these patterns on the cortical surface (with warm colors indicating regions with above-average activity and cool colors indicating regions with below-average activity, masked to show only those voxels that exhibited schema-related effects in Fig. 2b), we observe that the spatial patterns are highly reliable across the split halves, and the patterns are visually distinct between the two schemas.

Figure 5.

Effect of scrambling events on schematic correlations. A separate control group of 13 subjects was shown the same clips as in the main experiment, with the same schematically blocked structure, but drawn randomly from different stories and in a random order. We compared the magnitude of the schema effect for story pair correlations between subjects in the main experiment (as in Fig. 2) with the effect for correlations between subjects in the main experiment and the control experiment. Scrambling the event clips did not significantly disrupt schematic patterns in posterior regions of the cortex but did significantly reduce the similarity between schematic events in mPFC, specifically in the ventral portion of mPFC (inset). This result shows that schematic patterns in mPFC are significantly enhanced by having intact, predictable script structure on the timescale of multiple minutes. Surface map thresholded at FDR < 0.05 within mPFC. *p < 0.05, by permutation test.

Figure 6.

Schema classification of a new story. a, Using the labeled events from 7 of the 8 stories (from each schema), we computed the average spatial activity pattern for each of the 4 events (from each schema) and used these as the latent event representations in two HMMs. We then used the HMMs to find the best possible alignment of the held-out stories to each schema (without being given any event annotations) and predicted the schema of the held-out stories based on which of these alignments was better. b, SFG, mPFC, and PMC, and mPFC all showed classification rates well above chance, indicating that these regions exhibit robust schema-related patterns that can be identified in novel stories even without explicit event labels. **p < 0.01, by bootstrap test. Shaded regions represent bootstrap confidence distributions.

Figure 7.

Unsupervised alignment of events across stories. a, For each set of 8 stories within the same schema, we trained an HMM to identify a sequence of shared latent event patterns that was common across all the stories (without using any hand-labeled events). This correspondence can then be compared with the hand-labeled correspondence between stories. b, We can recover the schematic structure shared across stories using only fMRI data, without event labeling, from all three ROIs showing strong schema representations in the previous analyses (SFG, mPFC, and PMC). This match is closest when the number of latent events in the model is similar to the true number of events (4). Dotted line indicates p < 0.01 by permutation test, Bonferroni corrected for 5 tests. c, For example, the data-driven correspondence between time points of How I Met Your Mother and Up in the Air (from SFG, with 4 events) captures the shared schematic events (entering the airport, going through security, waiting at the boarding gate, and taking a seat on the plane) despite large differences in the details of the narratives.