Predictability modulates neurocognitive semantic processing of non-verbal narratives

Abstract

Predictability is known to modulate semantic processing in language, but it is unclear to what extent this applies for other modalities. Here we ask whether similar cognitive processes are at play in predicting upcoming events in a non-verbal visual narrative. Typically developing adults viewed comics sequences in which a target panel was highly predictable ("high cloze"), less predictable ("low cloze"), or incongruent with the preceding narrative context ("anomalous") during EEG recording. High and low predictable sequences were determined by a pretest where participants assessed "what happened next?", resulting in cloze probability scores for sequence outcomes comparable to those used to measure predictability in sentence processing. Through both factorial and correlational analyses, we show a significant modulation of neural responses by cloze such that N400 effects are diminished as a target panel in a comic sequence becomes more predictable. Predictability thus appears to play a similar role in non-verbal comprehension of sequential images as in language comprehension, providing further evidence for the domain generality of semantic processing in the brain.

Figure 1

Examples of high-cloze (A) and low-cloze (B) visual narrative stimuli. Bolded boxes indicate the critical panel in each sequence. In high-cloze stimuli, the critical panel is highly predictable given the prior context; in low-cloze stimuli, the critical panel is less predictable. Panel (C) shows an example of an anomalous sequence, in which the critical panel is completely semantically incongruent with the preceding context.

Figure 2

Illustration of the nine electrode clusters used for EEG analysis, with centers of each cluster labeled.

Figure 3

(A) ERPs at the critical panel for each condition at nine clusters across the scalp. Negativity is plotted upwards. Data is filtered using a 30 Hz lowpass filter for presentation purposes only. N300 and N400 effects are labeled. (B) Topographic plots of the three comparisons of interest across the five time windows of interest.

Figure 4

Correlations of cloze rating with amplitude. (A) Correlations of cloze rating with amplitude at each timepoint and electrode cluster. Correlation coefficients are plotted as a heat map with positive correlations in red. Non-significant correlation coefficients (as determined by cluster mass permutation tests to account for multiple comparisons) are masked in green. (B) Topographic plot of r correlation coefficients between cloze rating and amplitude from 500–600 ms at every electrode over the scalp. Black dots indicate electrodes where p < 0.05. (C) Scatterplot of cloze rating with amplitude at the C4 (left) and P4 (right) clusters from 500–600 ms. Negative is plotted upwards.