The differing impact of multisensory and unisensory integration on behavior

Abstract

Pooling and synthesizing signals across different senses often enhances responses to the event from which they are derived. Here, we examine whether multisensory response enhancements are attributable to a redundant target effect (two stimuli rather than one) or if there is some special quality inherent in the combination of cues from different senses. To test these possibilities, the performance of animals in localizing and detecting spatiotemporally concordant visual and auditory stimuli was examined when these stimuli were presented individually (visual or auditory) or in cross-modal (visual-auditory) and within-modal (visual-visual, auditory-auditory) combinations. Performance enhancements proved to be far greater for combinations of cross-modal than within-modal stimuli and support the idea that the behavioral products derived from multisensory integration are not attributable to simple target redundancy. One likely explanation is that whereas cross-modal signals offer statistically independent samples of the environment, within-modal signals can exhibit substantial covariance, and consequently multisensory integration can yield more substantial error reduction than unisensory integration.

Figure 1.

The apparatus and task. The orientation/approach task was performed in a 90-cm- diameter perimetry apparatus containing a complex of LEDs and speakers at 15° increments from 90° to the left and right of a central fixation point (0°). Each complex consisted of three LEDs and two speakers, each separated by 4 cm. In the present experiments, only the two outermost LEDs at any location were used. Trials consisted of randomly interleaved modality-specific (visual or auditory), cross-modal (visual–auditory), and within-modal (visual–visual or auditory–auditory) stimuli at each of the seven perimetric locations between ±45° (−45°, −30°, −15°, 0°, +15°, +30°, +45°), as well as catch trials.

Figure 2.

Multisensory integration was distinct from unisensory visual–visual integration. A, At every spatial location, multisensory integration produced substantial performance enhancements (94–168%; mean, 137%), whereas unisensory visual integration produced comparatively modest enhancements (31–79%; mean, 49%). Asterisks indicate comparisons that were significantly different (χ² test; p < 0.05). B, The pie charts to the left show performance in response to the modality-specific auditory (A₁) and visual (V₁ and V₂ are identical; see Materials and Methods) stimuli. The figures within the bordered region show the performance to the cross-modal (V₁A₁) and within-modal (V₁V₂) stimulus combinations. No-Go errors (NG; gray) and Wrong Localization errors (W; white) were significantly decreased as a result of multisensory integration, but only No-Go errors were significantly reduced as a result of unisensory integration. C, The differential effect of multisensory and unisensory integration was reasonably constant, regardless of the effectiveness of the best component stimulus and both showed an inverse relationship, wherein benefits were greatest when the effectiveness of the component stimuli was lowest. V, Visual; A, auditory; C, correct.

Figure 3.

Multisensory integration was distinct from unisensory auditory–auditory integration. Conventions are the same as for Figure 2. A, Multisensory integration provided significantly greater average performance (141% more) benefits than did unisensory auditory integration (49% more) at every perimetric location tested. B, Similarly, multisensory integration also significantly reduced both No-Go errors (NG; 60% less; gray) and Wrong Localization errors (W; 25% less; white), whereas unisensory auditory integration reduced only No-Go errors (28% less; gray). C, The differential effect of multisensory (V₁A₁) and unisensory (A₁A₂) integration was reasonably constant regardless of the effectiveness of the best component stimulus and both showed an inverse relationship; that is, benefits were greatest when the effectiveness of the component stimuli was lowest. V, Visual; A, auditory; C, correct.

Figure 4.

Multisensory integration was distinct from unisensory integration. The results of Experiment 1 (visual–visual vs visual–auditory integration) and Experiment 2 (auditory–auditory vs visual–auditory integration) were highly consistent. The magnitude of multisensory enhancement was approximately the same in the two experiments, whereas they were conducted at different times. Similarly, unisensory visual and unisensory auditory integration yielded approximately the same performance products.