Phase-encoded fMRI tracks down brainstorms of natural language processing with subsecond precision

Abstract

Natural language processing unfolds information overtime as spatially separated, multimodal, and interconnected neural processes. Existing noninvasive subtraction-based neuroimaging techniques cannot simultaneously achieve the spatial and temporal resolutions required to visualize ongoing information flows across the whole brain. Here we have developed rapid phase-encoded designs to fully exploit the temporal information latent in functional magnetic resonance imaging data, as well as overcoming scanner noise and head-motion challenges during overt language tasks. We captured real-time information flows as coherent hemodynamic waves traveling over the cortical surface during listening, reading aloud, reciting, and oral cross-language interpreting tasks. We were able to observe the timing, location, direction, and surge of traveling waves in all language tasks, which were visualized as "brainstorms" on brain "weather" maps. The paths of hemodynamic traveling waves provide direct evidence for dual-stream models of the visual and auditory systems as well as logistics models for crossmodal and cross-language processing. Specifically, we have tracked down the step-by-step processing of written or spoken sentences first being received and processed by the visual or auditory streams, carried across language and domain-general cognitive regions, and finally delivered as overt speeches monitored through the auditory cortex, which gives a complete picture of information flows across the brain during natural language functioning. PRACTITIONER POINTS: Phase-encoded fMRI enables simultaneous imaging of high spatial and temporal resolution, capturing continuous spatiotemporal dynamics of the entire brain during real-time overt natural language tasks. Spatiotemporal traveling wave patterns provide direct evidence for constructing comprehensive and explicit models of human information processing. This study unlocks the potential of applying rapid phase-encoded fMRI to indirectly track the underlying neural information flows of sequential sensory, motor, and high-order cognitive processes.

Keywords: brainstorms; dual-stream models; hemodynamic traveling waves; information flows; logistics models.

FIGURE 1

Experimental setup. (a) Subject wearing a “The Phantom of the Opera” style mask and a pair of noise‐canceling headphones, with resin clay filled around the head and headphones to reduce head motion and scanner noise. (b) A microphone in front of the subject's mouth. (c) A rear‐view mirror mounted above the 32‐channel head coil. (d) Acoustic foams placed inside the bore for scanner noise reduction. (e) MR‐compatible LCD monitor and eye tracker behind the scanner. (f) A close‐up view of the eye tracker.

FIGURE 2

Experimental paradigms and stimuli. (a) Reading only tasks, corresponding to Figure 4a. (b) Listening only tasks, corresponding to Figure 4b. (c) Reading‐aloud tasks, corresponding to Figure 4c. (d) Shadowing tasks, corresponding to Figure 4d. (e) Reading‐memorizing‐reciting and reading‐memorizing‐translating tasks, corresponding to Figures 4e and 5a. (f) Listening‐memorizing‐reciting and listening‐memorizing‐translating tasks, corresponding to Figures 4f and 5b,c. In (a–d), visual or auditory stimuli were presented between 0 and 5 s (Phase 1), followed by a blank screen between 5 and 16 s (Phase R). In (e,f), visual or auditory stimuli were presented between 0 and 5 s (Phase 1), followed by speech output between 5 and 10 s (Phase 2) and a blank screen between 5 and 16 s (Phase R).

FIGURE 3

Comparing phase distributions between brain regions. (a) A single‐subject phase‐encoded activation map of a Chinese reading‐memorizing‐reciting task. Five surface‐based regions of interest (sROIs) were selected based on sorted phases (1: Medial prefrontal cortex; 2: visual areas V4v and V8; 3: ventral premotor cortex; 4: inferior parietal lobe; 5: primary sensorimotor cortex). (b) Raw and average blood oxygen level dependent (BOLD) time courses over the scan duration (256 s) for voxels enclosed in each sROI. (c) Average BOLD signal change over one cycle (16 s) for each sROI. Each curve corresponds to same‐color time courses in (b). (d) Surge profile of each sROI where the y‐axis indicates the magnitude of activations (statistical significance), and the x‐axis indicates the phase within a cycle [0–2π] that corresponds to a time delay between 0 and 16 s. (e) Distribution of activation amplitudes and phases in the complex plane for five sROIs.

FIGURE 4

Group‐average maps (n = 21) of periodic activations in single‐language tasks. (a) Silent reading (0–5 s). (b) Listening (0–5 s). (c) Reading aloud (0–5 s). (d) Shadowing (listening to and immediately repeating a speech within 0–5 s). (e) Reading and memorizing a written sentence (0–5 s) then reciting it (5–10 s). (f) Listening to and memorizing a spoken sentence (0–5 s) then reciting it (5–10 s). Significant periodic activations are displayed on the inflated (lateral and ventral) and flattened cortical surfaces of a single subject. L1: Chinese; L2: English. LH, left hemisphere; RH, left hemisphere. The colorbar indicates the phase angles (0–2π) of blood oxygen level dependent signals corresponding to a trial period (0–16 s). A lower statistical significance threshold (F _[2,230] > 4.7; equivalent p < .01, uncorrected) is set for the maps of unimodal (reading and listening) tasks to match the spatial extent of significant activations (F _[2,230] > 9.6; equivalent p < .0001, uncorrected) in multimodal and crossmodal tasks. Color contours indicate regions activated by silent reading (cyan), listening only (yellow), and speaking (black). Solid contours: L1 activations; dotted contours: L2 activations.

FIGURE 5

Group‐average maps (n = 21) of periodic activations in translation tasks. (a) Text‐to‐speech translation (reading and memorizing a written sentence in 0–5 s, then verbalizing it in the target language in 5–10 s). (b) Speech‐to‐speech translation (listening to and memorizing a sentence in 0–5 s, then vocalizing it in the target language in 5–10 s). (c) Digit interpreting (listening to and memorizing five digits in 0–5 s, then vocalizing them in the target language in 5–10 s). All maps are displayed with a statistical significance F _[2,230] > 9.6 (equivalent p < .0001, uncorrected). LH, left hemisphere; RH, left hemisphere. L1: Chinese; L2: English. Solid and dotted contours represent unimodal L1 and L2 activations outlined in Figure 4.

FIGURE 6

Tracking the paths of traveling waves on isophase contour maps. (a) L2 reading task (Figure 4a, right). (b) L2 listening task (Figure 4b, right). (c) L1 reading‐memorizing‐reciting task (Figure 4e, left). (d) L1 (listening) to L2 (verbalizing) interpreting task (Figure 5b, left). (e–p) Each frame sequence shows the traveling waves (cf. storm rainbands) in a corresponding box in (a–d). The isophase of traveling waves in each frame is color‐coded and displayed over dark “clouds” representing the extent of activations in (a–d). Arrows in the leftmost frame of each sequence represent the paths of local traveling waves. See Figure S5–S22 for high‐resolution vector field maps.

FIGURE 7

Tracking traveling waves and surges on multilayer brain “weather” maps. (a) Topological maps with dark gray contours indicating the borders of visual, auditory and somatomotor areas displayed on the flattened cortical surfaces of a single subject (Table 2; see Sereno et al.,  for a full list of brain areas). (b) Multilayer single‐language activation maps overlaid on topological areas. Color contours (L1) and filled regions (L2) outline the extent of unimodal (reading and listening) or multimodal (speaking and self‐monitoring) activations in Figure 4. Black arrows indicate multiple streams of traveling waves in visual and auditory cortices. (c) Tracking the paths (yellow arrows) of regional traveling waves over topological areas (white contours) during a L1 reading‐memorizing‐reciting task (Figure 4e, left; see Figure S9 for a high‐resolution vector field map). Dark “clouds” represent the extent of significant periodic activations (F _[2,230] > 7.1; equivalent p < .001, uncorrected). Magenta contours indicate surface‐based regions of interest (sROIs) of language and domain‐general cognitive regions outside or partially overlapping topological areas. AIC, anterior insular cortex; ATL, anterior temporal lobe; A1, primary auditory cortex; dlPFC, dorsolateral prefrontal cortex; dmFAF, dorsomedial frontal auditory field; FEF, frontal eye fields; FOPaud, frontal operculum auditory; IPL, inferior parietal lobe; ITG, inferior temporal gyrus; LIP+, lateral intraparietal areas; M‐I, primary motor cortex; PCu, precuneus; PMd, dorsal premotor cortex; PMv, ventral premotor cortex; pre‐SMA, pre‐supplementary motor area; Pu, putamen; PV, parietal ventral somatosensory area; PZ, polysensory zone; P32aud, area p32, auditory part; Resp, respiratory area; SMA, supplementary motor area; STS, superior temporal sulcus; Spt, Sylvian parietal temporal area; S‐I, primary somatosensory cortex; rMFG, rostral middle frontal gyrus; VIP+, ventral intraparietal areas; V1+/V1−, V3A, V8, V4v, visual areas; 43aud, subcentral auditory area; 45aud, auditory area 45.

FIGURE 8

Surge profiles of traveling waves in six bilateral surface‐based regions of interest (sROIs; rostral middle frontal gyrus [rMFG], dorsal lateral prefrontal cortex [dlPFC], and inferior parietal lobe [IPL]). Each row corresponds to a task in Figures 4 and 5 as labeled. The height of surge is indicated by −log₁₀(uncorrected p‐value) in the y‐axis. Red and blue profiles indicate L1 and L2 single‐language tasks, respectively. Green and brown profiles indicate L1‐to‐L2 and L2‐to‐L1 translation tasks, respectively. L1: Chinese; L2: English. LH, left hemisphere; RH, left hemisphere.

FIGURE 9

Surge profiles of traveling waves in six bilateral surface‐based regions of interest (dorsomedial frontal auditory field [dmFAF], anterior insula cortex [AIC], and Sylvian parietal temporal area [Spt/PSaud]). All conventions are the same as those in Figure 8.

FIGURE 10

Surge profiles of traveling waves in six bilateral surface‐based regions of interest (anterior temporal lobe [ATL], superior temporal sulcus [STS], and inferior temporal gyrus [ITG]). All conventions are the same as those in Figure 8.

FIGURE 11

Logistics models. (a) A flowchart for logistics activities. Icons have free licenses and were downloaded from Flaticon (https://www.flaticon.com/). (b) A logistics model for language processing where sentence input mirrors the raw materials stage in (a); chunking a string of information for encoding and memory mirrors the packing, shipping, and processing stages in (a); converting information in working memory to speech and motor planning mirror the storage and route planning stages in (a); and speech output and self‐monitoring mirror the delivery and customer feedback stages in logistics activities.