Topographic mapping of a hierarchy of temporal receptive windows using a narrated story

Abstract

Real-life activities, such as watching a movie or engaging in conversation, unfold over many minutes. In the course of such activities, the brain has to integrate information over multiple time scales. We recently proposed that the brain uses similar strategies for integrating information across space and over time. Drawing a parallel with spatial receptive fields, we defined the temporal receptive window (TRW) of a cortical microcircuit as the length of time before a response during which sensory information may affect that response. Our previous findings in the visual system are consistent with the hypothesis that TRWs become larger when moving from low-level sensory to high-level perceptual and cognitive areas. In this study, we mapped TRWs in auditory and language areas by measuring fMRI activity in subjects listening to a real-life story scrambled at the time scales of words, sentences, and paragraphs. Our results revealed a hierarchical topography of TRWs. In early auditory cortices (A1+), brain responses were driven mainly by the momentary incoming input and were similarly reliable across all scrambling conditions. In areas with an intermediate TRW, coherent information at the sentence time scale or longer was necessary to evoke reliable responses. At the apex of the TRW hierarchy, we found parietal and frontal areas that responded reliably only when intact paragraphs were heard in a meaningful sequence. These results suggest that the time scale of processing is a functional property that may provide a general organizing principle for the human cerebral cortex.

Figure 1.

Examples of stimuli. A schematic of the stimuli used in the experiment. Forward (brown), backward, and scrambled (scram) versions of a real-life, 7 min story were used. To generate scrambled versions of the story, natural segments, defined by the end points of each word, sentence, and paragraph, were randomly shuffled at each of the following time scales: words (yellow), sentences (green), and paragraphs (blue).

Figure 2.

Effect of scrambling at different time scales. Maps of the reliability of responses across subjects were computed separately for each condition and superimposed on an inflated brain shown in lateral and medial views. The maps illustrate the extent of the reliable responses at sequentially higher levels of temporal coherence [backward story (A), word scrambled (B), sentence scrambled (C), paragraph scrambled (D), and intact forward story (E)] and each map subsumes the borders of the significantly reliable responses at the previous level. LS, Lateral sulcus; STS, superior temporal sulcus; CS, central sulcus; IPS, intraparietal sulcus; A, anterior; P, posterior.

Figure 3.

Hierarchical topography of temporal receptive windows. Response reliability (inter-SC) to auditory narratives as a function of temporal structures average across all subjects. Circled numbers correspond to graphs in Fig. 5. A voxel was assigned the label “backward” (red) when it was significant in all inter-SC maps (Fig. 5, ROI 1). A voxel was assigned the label “word scram” (yellow) when it was significant in all inter-SC maps except backward (Fig. 5, ROI 2). Similarly, a voxel was assigned the label “sentence scram” (green) when it was significant in all inter-SC maps except word scram and backward (Fig. 5, ROI 3). A voxel was assigned the label “paragraph scram” (blue) when it was significant only in the paragraph scrambled inter-SC maps (Fig. 5, ROI 4). Circled numbers indicate the approximate location of the ROIs along the A1-TPJ axis used in the ROI analysis. LS, Lateral sulcus; STS, superior temporal sulcus; CS, central sulcus; IPS, intraparietal sulcus; A, anterior; P, posterior; scram, scrambled.

Figure 4.

Hierarchical topography of temporal receptive windows in single subjects. Response reliability (inter-SC) to auditory narratives as a function of temporal structures for seven individual subjects, identified by initials. Reliability is assessed by comparing responses of each individual against the average of the responses of the remaining individuals. Figure layout is identical to that in Figure 3. LS, Lateral sulcus; STS, superior temporal sulcus; CS, central sulcus; IPS, intraparietal sulcus; A, anterior; P, posterior; scram, scrambled.

Figure 5.

Activation profiles from the ROIs along the A1–TPJ axis. Inter-SC shown as a function of temporal disruption of the auditory story in several independently defined ROIs. ROIs correspond to circled numbers in Figure 3. Early auditory areas (A1+, ROI 1) exhibited short TRWs, responding reliably to all conditions regardless of the level of temporal scrambling. Areas adjacent to A1+ along the superior temporal gyrus exhibited an intermediate TRW. Here, coherent information at the words (ROI 2), sentences (ROI 3), or longer time scales was necessary to evoke reliable activity. The longest TRWs within this axis were found in the posterior superior temporal sulcus and the TPJ. In these regions, reliable responses were evoked only by the paragraphs condition and the intact story condition (ROIs 4 and 5). FS, Forward story; P, paragraph; S, sentence; W, word; B, backward.

Figure 6.

Comparison of responses to the unscrambled (reordered) story with responses to the intact story. Moment-to-moment responses in A1+ evoked by the intact story are similar to the unscrambled backward, unscrambled sentence, and unscrambled paragraph response time courses. In area 3, the time courses to the intact story temporally resemble the unscrambled sentence and unscrambled paragraph time courses. In area 4, only the unscrambled paragraph time course is correlated with the intact story. These findings indicate that the responses within each processing stage are predominantly determined by the temporal structure within its preferred temporal window and are largely insensitive to longer temporal structure.

Figure 7.

TRW in the high-order mPFC and precuneus. A, Inter-SC correlation in the precuneus and mPFC evoked by forward story. B, Time course correlation analysis is identical to that shown in Figure 5. C, The results of the unscrambling procedure, as applied in areas 1–4 (Fig. 6). In the precuneus, only the unscrambled paragraph time course is correlate with the intact story, confirming that the time scale of processing in this area is relatively long (±30 s). In the mPFC, the reliable responses during the intact story did not correlate with unscrambled patterns from any of the other conditions. P, Paragraph; S, sentence; W, word; B, backward; FS, Forward story; UnP, unscrambled paragraphs; UnS, unscrambled sentences; rB, reversed backward; A, anterior; P, posterior; SAG, sagittal.

Figure 8.

An integrated view of the proposed hierarchy of time scales in visual and auditory modalities. A, The hierarchical temporal topography in the auditory domain. B, The hierarchical temporal topography in the visual domain [adapted from Hasson et al. (2008)]. C, Overlay of the results from A and B. The yellow line indicates the overlap, which occurs mostly in BA 39 and BA 40, and partially in lateral BA 7 and posterior BA 22. A, anterior; P, posterior.