The Human Brain Encodes a Chronicle of Visual Events at Each Instant of Time Through the Multiplexing of Traveling Waves

Abstract

The human brain continuously processes streams of visual input. Yet, a single image typically triggers neural responses that extend beyond 1s. To understand how the brain encodes and maintains successive images, we analyzed with electroencephalography the brain activity of human subjects while they watched ∼5000 visual stimuli presented in fast sequences. First, we confirm that each stimulus can be decoded from brain activity for ∼1s, and we demonstrate that the brain simultaneously represents multiple images at each time instant. Second, we source localize the corresponding brain responses in the expected visual hierarchy and show that distinct brain regions represent, at each time instant, different snapshots of past stimulations. Third, we propose a simple framework to further characterize the dynamical system of these traveling waves. Our results show that a chain of neural circuits, which each consist of (1) a hidden maintenance mechanism and (2) an observable update mechanism, accounts for the dynamics of macroscopic brain representations elicited by visual sequences. Together, these results detail a simple architecture explaining how successive visual events and their respective timings can be simultaneously represented in the brain.SIGNIFICANCE STATEMENT Our retinas are continuously bombarded with a rich flux of visual input. Yet, how our brain continuously processes such visual streams is a major challenge to neuroscience. Here, we developed techniques to decode and track, from human brain activity, multiple images flashed in rapid succession. Our results show that the brain simultaneously represents multiple successive images at each time instant by multiplexing them along a neural cascade. Dynamical modeling shows that these results can be explained by a hierarchy of neural assemblies that continuously propagate multiple visual contents. Overall, this study sheds new light on the biological basis of our visual experience.

Keywords: EEG; decoding; dynamical system; streams; time; visual perception.

Figure 1.

Successive images are simultaneously represented in brain activity. A, Visual contents refer to what is in the image at time (t). Visual flow refers to the amount of change between t₁ and t₂. B, Subjects watched ∼5000 randomly oriented Gabor patches, flashed every 250 ms and grouped into 8-item sequences separated by masks. Each sequence ended in a two-alternative forced choice where subjects indicated whether stimuli fell, on average, closer to the cardinal or diagonal axes. C, Brain activity was recorded with EEG. Each line shows the average response evoked by the stimuli. D, Top, Distribution of single-trial decoding error of Stim_n as a function of time relative to the onset of stimulus n. Bottom, Time course of the corresponding decoding score. The shaded regions (with an asterisk) indicate significant decoding across subjects (cluster corrected). E, Decoding scores of visual flows (approximated as the absolute angular difference between successive stimuli) as a function of time relative to stimulus onset. F, Cumulative decoding scores (black) and the contribution of each of the eight successive stimuli (color-coded by position in the 8-item sequence), as a function of time relative to the sequence onset (chance = 0). G, Similar to F for cumulative δ decoding scores. In panels C–G, the vertical lines mark the onsets of each stimulus. Error bars indicate the SEM across subjects.

Figure 2.

Visual representations propagate from sensory to associative cortices. A, Correlation scores resulting from encoding analyses, trained to predict the EEG activity from the sine and cosine of the stimulus angles. B, Each dot corresponds to a source estimated from the EEG coding topographies with a minimum norm estimation. The x-axis corresponds to the source location along the posteroanterior direction. Top, The y-axis either indicates the relative timing of the peak activity in each source or Bottom, The intensity of this peak. Asterisks indicate statistical significance (**p < 0.01, ***p < 0.001) C, Same data as B but plotted on the cortical surface. Colors indicate both the peak amplitude (e.g., black: amplitude = median amplitude across sources) and the peak latency (e.g., blue: peak within analogous to the 5% percentile of the earliest responses across sources; red: peak beyond 95% percentile). D, Correlation coefficients between deltas and EEG amplitude. E, F, Analogous analyses tp B-C applied to the brain responses coding for the changes between successive stimuli (δ). G, Cross-validated encoding scores (Pearson's r) obtained with both angles (sin + cos) and deltas. Colors indicate EEG channels. The results can be visualized interactively at https://kingjr.github.io/chronicles/.

Figure 3.

Dynamical system framework. A, Left, The membrane potential (top) and the expected spike rate (bottom) of an AdexpIAF neuron in response to a sustained input (onset and offset indicated by the ticks); the dotted lines indicate the basal activity at rest. Middle, In an excitatory/inhibitory population of neurons stimulated with an input, the alignment of pyramidal dendrites leads their PSP to be detectable from distant electrodes, whereas the interneurons of PSP are not detectable with EEG. Right, A linear dynamical system composed of two units (x = observable; y = hidden) connected in a feedback loop can approximate adaptive neurons or Excitatory/Inhibitory (E/I) balance responses: x captures an observable variable (e.g., the electric field associated with spiking activity or pyramidal PSP), whereas y captures a hidden variable (e.g., ion currents associated with adaptation or inhibitory PSP). B, Columns illustrate the predictions of no, positive and negative, and feedback loop circuits, respectively. The top black line illustrates the activity of an observable unit (x) in response to a stimulus (onset and offset marked by ticks). Decoding scores of stimulus angles (black) and deltas (red) from the simulated population x tuned to stimulus angles. The asterisks highlight whether Stim_n can be decoded after its offset. The TG matrices correspond to the decoding scores of each decoder trained at time t and tested at all time samples. C, More complex networks can be generated by hierarchically connecting feedback loops with one another. Arrows indicate connections within or between the levels of such hierarchy. D, Top, Examples of plausible hierarchies, together with the dynamics of their observable units (x) at each hierarchical level (black lines) in response to a brief stimulus and Bottom, The corresponding temporal generalization matrices.

Figure 4.

The spatiotemporal dynamics of representations reveal an updating hierarchy. A, Examples of TG for angle (top) and δ (bottom) decoders trained at 100, 150, 200, and 250 ms after stimulus onset and tested across all time samples. The shaded areas indicate significant generalization, cluster corrected across subjects. Time annotations indicate the duration during which each decoder significantly generalized. B, Full TG matrices for angle (top) and δ (bottom) decoders. Blue areas indicate below-chance generalizations. C, TG scores for each of the eight successive stimuli. Colored areas indicate the above-chance generalizations, cluster corrected across subjects. D, Left, Grid-search analyses across architectures, connection weights (w) and activation functions (f) led to search among >1.5 billion possible hierarchical models. Middle, Each of them was tested on its ability to generate dynamics qualitatively similar to those obtained empirically: that is, characterized by onset and offset responses whose durations increased across levels. Right, Two architectures captured these dynamics with no more than four connections. The plain line illustrates a representative example of an observable unit (x). The dotted line illustrates a representative example of the hidden unit (y).