Visual Processing by Hierarchical and Dynamic Multiplexing

Abstract

The complexity of natural environments requires highly flexible mechanisms for adaptive processing of single and multiple stimuli. Neuronal oscillations could be an ideal candidate for implementing such flexibility in neural systems. Here, we present a framework for structuring attention-guided processing of complex visual scenes in humans, based on multiplexing and phase coding schemes. Importantly, we suggest that the dynamic fluctuations of excitability vary rapidly in terms of magnitude, frequency and wave-form over time, i.e., they are not necessarily sinusoidal or sustained oscillations. Different elements of single objects would be processed within a single cycle (burst) of alpha activity (7-14 Hz), allowing for the formation of coherent object representations while separating multiple objects across multiple cycles. Each element of an object would be processed separately in time-expressed as different gamma band bursts (>30 Hz)-along the alpha phase. Since the processing capacity per alpha cycle is limited, an inverse relationship between object resolution and size of attentional spotlight ensures independence of the proposed mechanism from absolute object complexity. Frequency and wave-shape of those fluctuations would depend on the nature of the object that is processed and on cognitive demands. Multiple objects would further be organized along the phase of slower fluctuations (e.g., theta), potentially driven by saccades. Complex scene processing, involving covert attention and eye movements, would therefore be associated with multiple frequency changes in the alpha and lower frequency range. This framework embraces the idea of a hierarchical organization of visual processing, independent of environmental temporal dynamics.

Keywords: alpha; attention sampling; hierarchical multiplexing; phase coding; saccades; theta.

Figure 1.

Flexible oscillatory multiplexing and phase coding allows for processing single and multiple objects in different frames. left: wide attentional spotlight. A natural scene is composed of one or multiple objects. Here, you are watching your cat playing in the garden between trees. Both the trees and the cat are composed of several attributes like stem and crown (trees) or head and paws (cat). We suggest that each of those attributes is reflected on the neuronal level by distinct bursts of gamma band activity. Each burst thereby is tightly linked to a distinct phase of alpha band activity, whereby the respective features are “ranked” along alpha’s phase gradient, where attributes exerting higher neuronal activity (e.g., due to higher levels of saliency, attention, etc.) are processed earlier in the alpha phase, because they are able to overcome the pulsed inhibition earlier in the cycle. This mechanism would be at play for grouping (within an alpha cycle) the different elements of an object, while taking into account the specificity of each attribute. Similarly, each separate object in turn is coded along the theta/delta phase gradient (again depending on the level of excitation). Alternatively, this change of excitability could also be caused by saccadic eye movement itself. right: narrow attentional spotlight. If the focus of attention is more narrow (i.e., “zoomed in” attentional spotlight), we suggest that the frame of reference shifts, such that former attributes can become objects that are in turn composed of smaller attributes, allowing for a higher level of detail. Here, the focus of attention shifted towards the head of the cat, which is threatened by a predatory bird. In general, the same principles of phase coding apply, but now the level of detail has increased (i.e., the cat’s eyes become attributes and the cat’s head the object). Note that dynamic fluctuations are represented as sinusoidal oscillations only for illustrative purposes.

Figure 2.

Evidence supporting the framework a. The exact frequency of cortical oscillations in anticipation of the stimulus depends on task instructions (from Wutz et al., 2018). Participants were presented with two consecutive frames, separated by an inter stimulus interval (ISI). Each frame would display multiple graphical elements, and subjects were asked to either report an element absent in both frames (missing-element task or MET) or an element that was displayed in one half during the first frame and the other half during the second frame (odd-element task or OET). While the MET required cross-frame integration, the OET required cross-frame segregation. The authors showed that during the MET, the observed cortical frequency response (in the alpha range) was significantly lower in frequency as compared to the OET. This implies that the frequency of slow oscillations could be top-down controlled in order to favor integration of stimuli (with slower frequency) or segregation of the stimuli (with higher frequency). b. The attentional spotlight samples spatial locations at alpha frequency (from Gaillard et al., 2020). In a 100% validly cued task, monkeys had to detect a target randomly presented at the cued location following the cue (with distractors presented in some trials). The spatial position of the attentional spotlight, decoded from the multi-unit activity in both the x and y directions, exhibited an 8–12 Hz rhythm. c. Target detection times follow an 8 Hz behavioral rhythm within the same object and a 4 Hz rhythm between objects, indicating different sampling frequencies for within and between object sampling (from Fiebelkorn et al., 2013). After central fixation, a cue (75% validity) indicated to the participants that a target could appear at the cued location. In the remaining 25% of cases, the target could instead appear at a different location within the same object or within a different object. The distance to the original cued condition however would be equal. The detrended time course of the visual-target detection revealed a rhythmic profile with peaks at 4 and 8 Hz for invalidly cued detections. A 90° phase offset at 8 Hz was observed between the detection of the target at the cued location and the detection of a target on the same object, and an 180° offset at 4 Hz between the detection of the target at the cued location and on a different object. d. Behavioral oscillations in the alpha band are nested within theta cycles (from Song et al., 2014). After central fixation, an irrelevant pre-cue (used to reset attention) and a varying stimulus onset asymmetry (SOA), a target (either a circle or a square) was presented with equal chance within either of two peripheral boxes. Participants were asked to respond to the type of target (square or circle). The behavioral response time course depending on the SOA followed a complex oscillatory pattern of alpha oscillations nested in theta cycles (alpha power fluctuates in a theta rhythm).