A model of the temporal dynamics of multisensory enhancement

Abstract

The senses transduce different forms of environmental energy, and the brain synthesizes information across them to enhance responses to salient biological events. We hypothesize that the potency of multisensory integration is attributable to the convergence of independent and temporally aligned signals derived from cross-modal stimulus configurations onto multisensory neurons. The temporal profile of multisensory integration in neurons of the deep superior colliculus (SC) is consistent with this hypothesis. The responses of these neurons to visual, auditory, and combinations of visual-auditory stimuli reveal that multisensory integration takes place in real-time; that is, the input signals are integrated as soon as they arrive at the target neuron. Interactions between cross-modal signals may appear to reflect linear or nonlinear computations on a moment-by-moment basis, the aggregate of which determines the net product of multisensory integration. Modeling observations presented here suggest that the early nonlinear components of the temporal profile of multisensory integration can be explained with a simple spiking neuron model, and do not require more sophisticated assumptions about the underlying biology. A transition from nonlinear "super-additive" computation to linear, additive computation can be accomplished via scaled inhibition. The findings provide a set of design constraints for artificial implementations seeking to exploit the basic principles and potency of biological multisensory integration in contexts of sensory substitution or augmentation.

Keywords: Cross-modal; Enhancement; Modeling; Multisensory; Temporal dynamics.

Fig. 1

Unisensory and multisensory responses from two exemplar neurons in the cat SC. Both neurons overtly respond to visual (V) and auditory (A) stimuli presented in isolation and in combination (VA). Top plots indicate the traces and responses corresponding to each tested stimulus condition (dots indicate impulses). The middle plots illustrate the instantaneous firing rate (calculated by convolving the impulse train with a Gaussian kernel, 5 ms standard deviation) traces for each stimulus condition as well as the sum of the unisensory traces (dotted line). The scale has been changed to provide better visualization of the behavior near response onset. Bottom plots illustrate the mean cumulative impulse count, a running tally of the number of impulses elicited by each stimulus on or before each moment in time (i.e., without any smoothing or transformation). In the exemplar in (A), the total multisensory response magnitude is statistically greater (p < 0.05) than the sum of the two unisensory response magnitudes. In the exemplar in (B), it is not statistically different than the predicted sum. However, both show similar nonlinear computations at the beginning of the response.

Fig. 2

Empirical findings of nonlinear enhancements in magnitude and timing in a population of multisensory SC neurons. (A) Comparison of the multisensory response magnitude (mean # stimulus-elicited impulses) to the largest component unisensory response magnitude (top) and to the sum (Σ) of the unisensory response magnitudes (bottom). (B) and (C) Average difference between the multisensory and summed unisensory instantaneous firing rates for all traces in the population, synchronized to the onset of the multisensory response (solid line = mean, dashed line = median) and plotted as a function of time (B) and total multisensory response duration (C). (D) Speed comparisons between the multisensory and summed unisensory responses. Plotted is the cumulative frequency distribution for the time difference between when the multisensory and summed unisensory responses first crossed each of two criterion levels: the peak firing rate of the weaker of the two (max) and a firing rate 1/2 in magnitude (1/2 max). Negative sample values indicate a faster multisensory response. (E) Left: bar graph comparing the relative incidence of multisensory responses peaking before the peak of the summed unisensory responses (multi. first) versus the reverse (Σ first). Right: schematic of the mean values for peak firing times and rates at those times for the population. Time values are relative to visual stimulus onset.

Fig. 3

Model architecture. (A) An example of an input trace that begins rising at time 0 and peaks at t = 40 ms (D = 0, λ₁ = 20 ms, I_max = 1, T = 40 ms, λ₂ = 100 ms). (B) An example of the membrane potential over time in a single trial of a neuron responding to the stimulus in (A), contaminated with a noise current having parameters (μ = 0, σ = 5). Vertical lines are drawn at times where action potentials are generated. The dotted line indicates the unit’s threshold. (C) Top: an impulse raster of 100 simulations having the same parameters as (B). Bottom: instantaneous firing rate of the above raster using a Gaussian kernel with 5 ms standard deviation.

Fig. 4

Model results. (A) Example multisensory (solid line) and summed unisensory (dotted line) response traces from a simulated multisensory neuron (D = 0, λ₁ = 20 ms, I_max = 1, T = 40 ms, λ₂ = 100 ms, μ = 0, σ = 5, τ = 10 ms, R = 2 ms, h = 0.5). The two unisensory signals contributing to the multisensory response were assumed to be exactly equal. The inset expands the activity in the window 0–50 ms. The multisensory response begins earlier, rises faster, and peaks earlier at a higher magnitude than predicted by the summed unisensory traces. (B) The difference between the multisensory and summed unisensory inputs for multiple simulations (solid = mean, dashed = median) with the parameters of (A). (C) Relationship between the multisensory and summed unisensory firing rates for model units (μ = 0, σ = 5, τ = 10 ms, R = 2 ms, h = 0) responding to constant input (0 < I < 5). The line of unity indicates an additive multisensory response. The two curves illustrate the relationship for circumstances in which the two unisensory inputs are equal in magnitude (a = 1, solid line) one input is 50% as strong as the other (a = 0.5). The inset shows the relationship between the firing rates for the multisensory (multi, with a = 1), summed unisensory (Σ), and unisensory (Uni) responses and the input magnitude. (D) The relationship between firing rate and input magnitude for three different values of the noise current standard deviation (σ).