Your Brain on Comics: A Cognitive Model of Visual Narrative Comprehension

Abstract

The past decade has seen a rapid growth of cognitive and brain research focused on visual narratives like comics and picture stories. This paper will summarize and integrate this emerging literature into the Parallel Interfacing Narrative-Semantics Model (PINS Model)-a theory of sequential image processing characterized by an interaction between two representational levels: semantics and narrative structure. Ongoing semantic processes build meaning into an evolving mental model of a visual discourse. Updating of spatial, referential, and event information then incurs costs when they are discontinuous with the growing context. In parallel, a narrative structure organizes semantic information into coherent sequences by assigning images to categorical roles, which are then embedded within a hierarchic constituent structure. Narrative constructional schemas allow for specific predictions of structural sequencing, independent of semantics. Together, these interacting levels of representation engage in an iterative process of retrieval of semantic and narrative information, prediction of upcoming information based on those assessments, and subsequent updating based on discontinuity. These core mechanisms are argued to be domain-general-spanning across expressive systems-as suggested by similar electrophysiological brain responses (N400, P600, anterior negativities) generated in response to manipulation of sequential images, music, and language. Such similarities between visual narratives and other domains thus pose fundamental questions for the linguistic and cognitive sciences.

Keywords: Comics; Discourse; Linguistic processing; Narrative; Visual language; Visual narrative.

Figure 1

Mechanisms operating over representational levels of both semantics and narrative in the processing of visual narrative sequences. Single‐headed arrows represent feedforward and feed‐backward connections within representational levels. Double‐headed arrows represent the interfaces between semantic and narrative processing for different stages.

Figure 2

Illustration of the semantic representational level for visual narratives, where a reader accesses the semantic information in images, which thereby is incorporated into a situation model of the elements and events of that scene.

Figure 3

Event‐related potentials to manipulation of narrative and/or semantic structures in two experiments: (a) the N400 elicited by semantic incongruity is insensitive to the presence of narrative structure (Cohn et al., 2012), while (b) an anterior negativity elicited by narrative patterning is insensitive to the presence of semantic incongruity, though the (c) P600 is modulated by both narrative and semantics (Cohn & Kutas, 2017). Each graph depicts one electrode site, with (a) being at the midline central point of the scalp (Cz), (b) being the midline prefrontal (MiPf), and (c) being the midline parietal (MiPf). The x‐axis depicts the time course of processing in milliseconds, while the y‐axis depicts amplitude, with negative up. Separation of waves indicates a difference in processing, with relevant epochs highlighted.

Figure 4

Visual sequences showing (a) canonical narrative schema in Visual Narrative Grammar, and (b) schema combined in hierarchic constituent structures.

Figure 5

(a) An example sequence with a well‐formed narrative structure but no semantic relations between panels, and (b) a diagram of its processing across narrative and semantic representational levels. Semantic cues provide adequate mappings to categories, which are correctly ordered into the narrative schema. Yet the activated semantic information maintains no relations between panels.

Figure 6

Modifying schema in VNG applied to Fig. 4a using (a) a conjunction schema which repeats narrative categories within a node (here Establishers) and (b) a head‐modifier schema which elaborates on a “head”—here an Initial modified by a Refiner.

Figure 7

Illustration of the narrative representational level of sequential images, where semantic cues in images map to categorical roles, which in term sponsor predictions based on a schematic order (in blue), and are thereby revised in the face of incoming information.