Integration of Motion and Form Cues for the Perception of Self-Motion in the Human Brain

Abstract

When moving around in the world, the human visual system uses both motion and form information to estimate the direction of self-motion (i.e., heading). However, little is known about cortical areas in charge of this task. This brain-imaging study addressed this question by using visual stimuli consisting of randomly distributed dot pairs oriented toward a locus on a screen (the form-defined focus of expansion [FoE]) but moved away from a different locus (the motion-defined FoE) to simulate observer translation. We first fixed the motion-defined FoE location and shifted the form-defined FoE location. We then made the locations of the motion- and the form-defined FoEs either congruent (at the same location in the display) or incongruent (on the opposite sides of the display). The motion- or the form-defined FoE shift was the same in the two types of stimuli, but the perceived heading direction shifted for the congruent, but not for the incongruent stimuli. Participants (both sexes) made a task-irrelevant (contrast discrimination) judgment during scanning. Searchlight and ROI-based multivoxel pattern analysis revealed that early visual areas V1, V2, and V3 responded to either the motion- or the form-defined FoE shift. After V3, only the dorsal areas V3a and V3B/KO responded to such shifts. Furthermore, area V3B/KO shows a significantly higher decoding accuracy for the congruent than the incongruent stimuli. Our results provide direct evidence showing that area V3B/KO does not simply respond to motion and form cues but integrates these two cues for the perception of heading.SIGNIFICANCE STATEMENT Human survival relies on accurate perception of self-motion. The visual system uses both motion (optic flow) and form cues for the perception of the direction of self-motion (heading). Although human brain areas for processing optic flow and form structure are well identified, the areas responsible for integrating these two cues for the perception of self-motion remain unknown. We conducted fMRI experiments and used multivoxel pattern analysis technique to find human brain areas that can decode the shift in heading specified by each cue alone and the two cues combined. We found that motion and form cues are first processed in the early visual areas and then are likely integrated in the higher dorsal area V3B/KO for the final estimation of heading.

Keywords: V3B/KO; cue integration; form; heading; motion; optic flow.

Figure 1.

An illustration of the animated Glass pattern stimulus that offers two independent FoEs: the form-defined FoE given by the orientation of the dot pairs (“×”) and the motion-defined FoE given by the motion of the dot pairs (“+”). Lines with arrowheads indicate velocity vectors of the centroid of the dot pairs. “×,” “+,” and lines with arrowheads are for illustration purpose only and not shown in the experimental stimulus.

Figure 2.

Experiment 1 visual stimuli and data. a, Illustrations of the six stimuli. Negative sign indicates the FoE location to the left of the display center; positive sign indicates the FoE location to the right. “×” and “+” indicate the form- and the motion-defined FoEs, respectively. b, The classifier's decoding accuracy for the six stimuli for each ROI group. Gray bars plot the data with 100 most activated voxels. White bars plot the data with 200 most activated voxels. (see also Figure 2–1) Dotted lines indicate the upper limits of the 95% CI of the classifier's baseline decoding accuracies from 1000 shuffled tests. Solid line indicates the chance level of 1/6. Error bars indicate SEs across 14 participants. ***p < 0.001. **p < 0.01. *p < 0.05. c, The searchlight brain map showing clusters (≥25 voxels) that have significantly higher decoding accuracies than the baseline levels across 14 participants (t₍₁₃₎ > 2.16, p < 0.05). d, The classifier's decoding accuracy as a function of the form-defined FoE position shift. Solid lines indicate the fitted linear functions. Error bars indicate SEs across 14 participants.

Figure 3.

Experiment 2 visual stimuli and data. a, Illustrations of the four stimuli. Negative sign indicates the FoE location to the left of the display center; positive sign indicates the FoE location to the right. “×” and “+” indicate the form- and the motion-defined FoEs, respectively. b, Data from the psychophysics experiment. Left: Mean percentage of “right” response in heading judgments as a function of the average location of the motion- and the form-defined FoEs. Solid lines indicate cumulative Gaussian functions fitted to the data averaged across participants. Right: Mean PSE against stimulus type. Error bars indicate SEs across 15 participants. c, The classifier's decoding accuracy for the congruent (gray) and incongruent (white) stimuli for the six visual areas that encode the form-defined FoE shift as identified in Experiment 1. Dotted lines indicate the upper limits of the 95% CI of the classifier's baseline decoding accuracies from 1000 shuffled tests. Solid line indicates the chance level of 1/2. Error bars indicate SEs across 13 participants. *p < 0.05.

Figure 4.

Visual stimuli and decoding accuracies for the motion- or the form-defined FoE shift. a, Illustrations of the stimuli with only the motion- or only the form-defined FoE shift. “×” and “+” indicate the form- and the motion-defined FoEs, respectively. b, The classifier's decoding accuracy for the motion- (white) or the form-defined FoE shift (gray) for the six visual areas identified in Experiment 1. Dotted lines indicate the upper limits of the 95% CI of the classifier's baseline decoding accuracies from 1000 shuffled tests. Solid line indicates the chance level of 1/2. Error bars indicate SEs across 13 participants.

Figure 5.

Sensitivity and integration index data. a, The motion cue (d_m^′), the form cue (d_f^′), and the combined cue (d_m+f^′) sensitivity indices for the six visual areas. Dotted lines indicate the quadratic sums of d_m^′ and d_f^′. Error bars indicate SEs across 13 participants. b, The integration indices for the six visual areas. Black line in the center of each bar indicates the median. Edges represent 68% CIs. Error bars indicate the 95% CIs. *p < 0.05.

Figure 6.

Experiment 3 visual stimuli and data. a, Illustrations of the stimuli with the form signals randomized and the motion signals intact (top row) and the stimuli with the motion signals randomized and the form signals intact (bottom row). Negative sign indicates the FoE location to the left of the display center; positive sign indicates the FoE location to the right. The “×” and the “+” indicate the form- and the motion-defined FoEs, respectively. b, The classifier's decoding accuracy for the form- (gray) or the motion-defined FoE shift (white) with the form-signal-randomized stimuli (left) and the motion-signal-randomized stimuli (right) for the six visual areas. c, The classifier's decoding accuracy for the congruent (gray) or incongruent (white) stimuli with the form-signal-randomized stimuli (left) and the motion-signal-randomized stimuli (right) for the six visual areas. Dotted lines indicate the upper limits of the 95% CI of the classifier's baseline decoding accuracies. Solid line indicates the chance level of 1/2. Error bars indicate SEs across 12 participants.

Figure 7.

Eye movement data. a, The proportion of eye position data of 6 participants as a function of the deviation between eye fixation and the center of the display along horizontal and vertical directions for the six stimuli in Experiment 1 (top) and the four stimuli in Experiment 2 (bottom). b, The saccade amplitude and the number of saccades against the six stimuli in Experiment 1 (top) and the four stimuli in Experiment 2 (bottom). Error bars indicate SEs across 6 participants.