Cortical Mechanisms of Multisensory Linear Self-motion Perception

Abstract

Accurate self-motion perception, which is critical for organisms to survive, is a process involving multiple sensory cues. The two most powerful cues are visual (optic flow) and vestibular (inertial motion). Psychophysical studies have indicated that humans and nonhuman primates integrate the two cues to improve the estimation of self-motion direction, often in a statistically Bayesian-optimal way. In the last decade, single-unit recordings in awake, behaving animals have provided valuable neurophysiological data with a high spatial and temporal resolution, giving insight into possible neural mechanisms underlying multisensory self-motion perception. Here, we review these findings, along with new evidence from the most recent studies focusing on the temporal dynamics of signals in different modalities. We show that, in light of new data, conventional thoughts about the cortical mechanisms underlying visuo-vestibular integration for linear self-motion are challenged. We propose that different temporal component signals may mediate different functions, a possibility that requires future studies.

Keywords: Multisensory integration; Optic flow; Self-motion perception; Vestibular.

Fig. 1

A virtual reality system for studying visuo-vestibular integration. A Schematic of the experimental paradigm. The motion platform and visual display provide inertial motion signals and optic flow signals, respectively, to simulate locomotion in the environment. The black arrow indicates the forward motion of the motion base, and θ indicates the deviation of motions from straight ahead. B Transient motion has a Gaussian-shaped velocity curve, with a biphasic acceleration profile to activate the peripheral vestibular channel. Optic flow follows the same profile. Redrawn using data from ref. [23]. C Average normalized psychophysical thresholds across behavioral sessions from four monkeys who discriminated heading directions based on vestibular alone (Ves), visual alone (Vis), and a combination of the two stimuli (Comb). “Pred” indicates the threshold predicted from the Bayesian optimal cue integration theory, which is computed based on thresholds in either single cue conditions (equation 1). In each trial during the heading discrimination task, animals experienced linear forward motion based on either type of stimulus with a small deviation (θ) of the leftward or rightward component. The animals were typically required to maintain central fixation across the stimulus duration (1–2 s). At the end of each trial, the central fixation point disappeared, providing a “go” signal. The animals made saccadic eye movements to one of the two choice targets presented on each side of the visual display to report their experienced heading direction. A correct choice would lead to a reward. Redrawn using data with permission from ref. [23, 32].

Fig. 2

Temporal dynamics of vestibular and visual responses across cortical areas. A Schematic of the MSTd in the extrastriate visual cortex, LIP in the parietal cortex, and FEF in the frontal cortex. B–D Time course of population average responses to leftward and rightward heading directions quantified by Δ firing rates (ΔPSTH) in the three areas. Experimental conditions were almost identical in these studies, yet the location of choice targets was a bit different across studies. In the MSTd, choice targets were always aligned in the horizontal meridian [32]. In the LIP [23] and FEF [24], choice targets were placed at locations matching the response field of the recorded neurons since these areas are more influenced by the preparation and execution of the saccadic response. Thus, different from the MSTd, the time-course of neuronal activity in the LIP and FEF typically reflects mixed signals of sensory, sensory accumulation, sensory-motor transformation (choice), and motor execution. Solid and dashed gray curves represent the acceleration and velocity profiles of the motion stimulus, respectively. Redrawn using data with permission from ref. [23, 24, 32]. MSTd, medial superior temporal sulcus; LIP, lateral intraparietal area; FEF, frontal eye field.

Fig. 3

Manipulating temporal offset for visuo-vestibular input. A Average psychophysical thresholds for two animals in the two unimodal and five bimodal stimuli conditions with different temporal offsets (ΔT; −750, −500, −250, 0, and 250 ms) between vestibular and visual inputs in the heading discrimination task that is described in the previous section (e.g., Fig. 1C). B From left to right: time course of population Fisher information in the FEF under different temporal offset conditions. Ves+Vis represents predictions from a straight sum algorithm based on the two single cue inputs. Comb represents the measured information under bimodal stimuli conditions. Redrawn using data with permission from ref. [24].

Fig. 4

Hypothetical neural circuits mediating visuo-vestibular heading perception. To estimate heading, vestibular acceleration and visual speed signals are hypothesized to propagate through individual pathways, for example, the PIVC and extrastriate visual cortex, respectively, to high-level decision-related areas including sites in the frontal lobe and posterior parietal lobe (late convergence/integration pathway). In these areas, each type of momentary sensory evidence is accumulated and can be described by a drift-diffusion model (see ref. [90]) (blue and red curves). Cross-modality sensory evidence is also integrated in these areas (green curve). Alternatively, the previously observed convergence of two signals in the sensory area MSTd (early convergence/integration pathway) may be involved in other contexts, for example, computing displacement (panel on the right side of the vertical dashed line). Briefly, vestibular and visual signals with speed temporal profiles converge and are integrated in the MSTd. These signals are then transmitted to decision-related areas for evidence accumulation and decision formation. FEFsac, saccadic area of the frontal eye field; LIP, lateral intraparietal area; MSTd, medial superior temporal sulcus; MT, middle temporal area; PIVC, parieto-insular vestibular cortex.