Multisensory integration: psychophysics, neurophysiology, and computation

Abstract

Fundamental observations and principles derived from traditional physiological studies of multisensory integration have been difficult to reconcile with computational and psychophysical studies that share the foundation of probabilistic (Bayesian) inference. We review recent work on multisensory integration, focusing on experiments that bridge single-cell electrophysiology, psychophysics, and computational principles. These studies show that multisensory (visual-vestibular) neurons can account for near-optimal cue integration during the perception of self-motion. Unlike the nonlinear (superadditive) interactions emphasized in some previous studies, visual-vestibular neurons accomplish near-optimal cue integration through subadditive linear summation of their inputs, consistent with recent computational theories. Important issues remain to be resolved, including the observation that variations in cue reliability appear to change the weights that neurons apply to their different sensory inputs.

Fig. 1

Statistically (Bayes)-optimal cue integration: theory and behavioral data (a), (b) Bayesian predictions for the relationships between probability distributions (top) and behavioral response functions (bottom). Green curves represent the situation in which two sensory cues are presented together, whereas red and blue curves represent the relevant quantities for each cue presented separately (red and blue). In (a), cues 1 and 2 have equal reliability (same variance); thus, the posterior distribution (green) is equally influenced by both cues. In (b), cue 2 has a larger variance (lower reliability) than cue 1. According to Bayes’ rule and assuming a uniform prior (i.e., all stimuli are equally probable), the resultant posterior distribution (green) has smaller variance than the likelihoods for each cue (red, blue) and its mean lies closer to the cue with higher reliability (red). Note that the behavioral response functions here (bottom) represent the integral of the posterior distributions under the assumption of Gaussian distributions and uniform priors. (c) Psychometric data and (d) average behavioral thresholds for macaques trained in a 2AFC heading discrimination task. Data are shown separately for vestibular (red), visual (blue) and bimodal stimuli (green). Thresholds were computed as the standard deviation of a cumulative Gaussian function fitted to the data (smooth curves). Performance in the bimodal condition closely matched Bayesian predictions (black). Note that, in these experiments, the reliability of the two cues was approximately matched by reducing visual motion coherence. Panels (c) and (d) are reprinted with permission from Gu Y, Angelaki DE, DeAngelis GC. Neural correlates of multisensory cue integration in macaque MSTd. Nat. Neurosci. 11: 1201-1210, 2008.

Fig. 2

Neural correlates of Bayesian cue integration. (a) Mean firing rates and (b) corresponding neurometric functions from a congruent MSTd neuron. Neuronal thresholds were computed as the standard deviation of a cumulative Gaussian function fitted to the neurometric data (smooth curves in b). (c) Neuronal thresholds under cue combination depend on the congruency of visual and vestibular tuning. The ratio of the threshold measured in the combined condition relative to the prediction from optimal cue integration is plotted as a function of Congruency Index (CI). CI ranges from +1 (when visual and vestibular tuning functions have a consistent slope) to -1 (when they have opposite slopes). Thus, congruent cells have CI significantly greater than zero (filled symbols), whereas opposite cells have a significnat CI<0 (open symbols); cells with intermediate tuning properties are shown with asterisks. Dashed horizontal line: threshold in the combined condition is equal to the prediction. (d) Choice probabilities, reflecting correlations between neural activity and perceptual decisions, also depend on congruency of tuning. Reprinted with permission from Gu Y, Angelaki DE, DeAngelis GC. Neural correlates of multisensory cue integration in macaque MSTd. Nat. Neurosci. 11: 1201-1210, 2008.