Multisensory integration in macaque visual cortex depends on cue reliability

Abstract

Responses of multisensory neurons to combinations of sensory cues are generally enhanced or depressed relative to single cues presented alone, but the rules that govern these interactions have remained unclear. We examined integration of visual and vestibular self-motion cues in macaque area MSTd in response to unimodal as well as congruent and conflicting bimodal stimuli in order to evaluate hypothetical combination rules employed by multisensory neurons. Bimodal responses were well fit by weighted linear sums of unimodal responses, with weights typically less than one (subadditive). Surprisingly, our results indicate that weights change with the relative reliabilities of the two cues: visual weights decrease and vestibular weights increase when visual stimuli are degraded. Moreover, both modulation depth and neuronal discrimination thresholds improve for matched bimodal compared to unimodal stimuli, which might allow for increased neural sensitivity during multisensory stimulation. These findings establish important new constraints for neural models of cue integration.

Fig. 1

Peri-stimulus time histograms (PSTHs) of neural responses for an example MSTd neuron. Gray curves indicate the Gaussian velocity profile of the stimulus. The two leftmost PSTHs show responses in the unimodal visual condition for the preferred and anti-preferred headings. Analogously, the two bottom PSTHs represent preferred and anti-preferred responses in the unimodal vestibular condition. PSTHs within the gray box show responses to bimodal conditions corresponding to the 4 combinations of the preferred and anti-preferred headings for the two cues. Dashed vertical lines bound the central 1s of the stimulus period, during which mean firing rates were computed.

Fig. 2

Examples of tuning for two “opposite” MSTd neurons. Color contour maps show mean firing rates as a function of vestibular and visual headings in the bimodal condition. Tuning curves along the left and bottom margins show mean (± SEM) firing rates versus heading for the unimodal visual and vestibular conditions, respectively. Data collected using visual stimuli (optic flow) having 100% and 50% motion coherence are shown in the left and right columns, respectively. (A) Data from a neuron with opposite vestibular and visual heading preferences in the unimodal conditions (same cell as in Fig. 1). Black dots indicate the bimodal response conditions corresponding to the PSTHs shown in Fig. 1. Bimodal tuning shifts from visually dominated at 100% coherence to balanced at 50% coherence. (B) Data from another “opposite” neuron. Bimodal responses reflect an even balance of visual and vestibular headings at 100% coherence, and become vestibularly dominated at 50% coherence. Inset: A top-down view showing the 8 possible heading directions (for each cue) in the horizontal plane.

Fig. 3

Data for a “congruent” MSTd cell, tested at three motion coherences. Format is the same as in Fig. 2. (A) Bimodal responses at 100% coherence are visually dominated. (B) Bimodal responses at 50% coherence show a balanced contribution of visual and vestibular cues. (C) At 25% coherence, bimodal responses appear to be dominated by the vestibular input.

Fig. 4

Fitting of linear and non-linear models to bimodal responses. (A–D) Model fits and errors for the same neuron as in Fig. 3. Color contour maps show fits to the bimodal responses using (A) a weighted sum of the unimodal vestibular and visual responses and (B) a weighted sum of the unimodal responses plus their product. (C), (D) Errors of the linear and non-linear fits, respectively. (E) Variance accounted for (VAF) from the non-linear fits is plotted against VAF from the linear fits. Data measured at 100% coherence are shown as circles; 50% coherence as triangles. Filled symbols represent neurons (16 of 44 neurons at 100% coherence and 8 of 14 at 50% coherence) whose responses were fit significantly better by the model including the product term (sequential F test, P<0.05).

Fig. 5

Dependence of vestibular and visual response weights on visual motion coherence. (A), (B) Histograms of vestibular and visual weights (linear model) computed from data at 100% (black) and 50% (gray) coherence. Triangles are plotted at the medians. (C), (D) Vestibular and visual weights (linear model) are plotted as a function of motion coherence. Data points are coded by the significance of unimodal visual tuning (open vs. filled circles) and by the congruency between unimodal vestibular and visual heading preferences (colors). (E) The ratio of visual to vestibular weights is plotted as a function of coherence. For each cell, this ratio was normalized to unity at 100% coherence. Errors bars show standard errors. Filled symbols and solid line: weights computed from linear model fits. Open symbols and dashed line: weights computed from non-linear model fits. (F) Comparison of variance-accounted-for (VAF) between linear models with yoked weights and independent weights. The filled data points (17 of 23 neurons) were fit significantly better by the independent weight model (sequential F test, P<0.05). Of 23 total neurons, 12 with unimodal tuning that remained significant (ANOVA P<0.05) at lower coherences are plotted as circles. Triangles indicate neurons that failed unimodal tuning criteria when coherence was reduced to 50% and/or 25%.

Fig. 6

Modulation depth and visual-vestibular congruency. (A) Illustration of “aligned” (magenta) and “anti-aligned” (orange) diagonals through the bimodal response array (color contour plot). (B) Mean (± SEM) firing rates for aligned and anti-aligned bimodal stimuli (extracted from the data in A). The anti-aligned tuning curve is plotted as a function of the visual heading. (C), (D) Modulation depth (maximum-minimum response) for bimodal aligned and anti-aligned stimuli is plotted against the largest unimodal (vestibular or visual) response modulation. Color indicates the congruency between vestibular and visual heading preferences (red: congruent cells; blue: opposite cells; black: intermediate cells).

Fig. 7

Fisher information and heading discriminability. (A) Example wrapped Gaussian fits to vestibular, visual, and “matched” tuning curves for the neuron shown in Fig. 2B. Error bars: ± SEM. (B) Population histogram of VAF for parametric fits to vestibular (blue), visual (green) and matched (magenta) tuning curves. Filled bars denote fits to 100% coherence data; open bars to 50% coherence data. (C), (D) Population vestibular (blue), visual (green) and matched (magenta) tuning curves for N=44 cells (100% coherence, C) and N=14 cells (50% coherence, D). Each curve was shifted to align the peaks of all tuning curves at 0° before averaging. (E) Fisher information versus heading. Using fits from A, Fisher information was computed as $I_F\left(\theta \right)=\frac{\mathrm{R\prime }{\left(\theta \right)}^2}{{\sigma }^2\left(\theta \right)},$, where R’(θ) is the derivative of the fitted tuning curve at a particular heading (θ), and σ²(θ) is the response variance. (F) Discrimination threshold versus heading for the same example neuron as in A and E. Thresholds were computed as $\mathrm{\Delta }\theta \left(\theta \right)=1/\sqrt{I_F\left(\theta \right)}.$ The two numerically computed minima for each curve are plotted as asterisks.

Fig. 8

Comparison of estimated heading discrimination thresholds for bimodal (matched) and unimodal (A, vestibular, B, visual) responses. Circles and triangles represent data collected at 100% and 50% coherence, respectively. Color indicates the congruency between unimodal heading preferences (red: congruent cells; blue: opposite cells; black: intermediate cells).