Stimulus value gates multisensory integration

Abstract

The brain enhances its perceptual and behavioral decisions by integrating information from its multiple senses in what are believed to be optimal ways. This phenomenon of "multisensory integration" appears to be pre-conscious, effortless, and highly efficient. The present experiments examined whether experience could modify this seemingly automatic process. Cats were trained in a localization task in which congruent pairs of auditory-visual stimuli are normally integrated to enhance detection and orientation/approach performance. Consistent with the results of previous studies, animals more reliably detected and approached cross-modal pairs than their modality-specific component stimuli, regardless of whether the pairings were novel or familiar. However, when provided evidence that one of the modality-specific component stimuli had no value (it was not rewarded) animals ceased integrating it with other cues, and it lost its previous ability to enhance approach behaviors. Cross-modal pairings involving that stimulus failed to elicit enhanced responses even when the paired stimuli were congruent and mutually informative. However, the stimulus regained its ability to enhance responses when it was associated with reward. This suggests that experience can selectively block access of stimuli (i.e., filter inputs) to the multisensory computation. Because this filtering process results in the loss of useful information, its operation and behavioral consequences are not optimal. Nevertheless, the process can be of substantial value in natural environments, rich in dynamic stimuli, by using experience to minimize the impact of stimuli unlikely to be of biological significance, and reducing the complexity of the problem of matching signals across the senses.

Keywords: behavior; cat; enhancement; orientation; plasticity.

FIGURE 1

Conceptual representation of the experiments designed to examine how experience with cross-modal cues affects multisensory performance. (a) One cross-modal pair (A1V1hi) was consistently presented together during training and was of high reward value (depicted here as a bird singing). During testing, this stimulus pair elicited a rapid, coordinated orientation response much more reliably than either component stimulus. (b) The other combination was never presented together during training. It involved a visual cue of modest reward value that elicited orientation when presented alone (V2low - depicted as an insect) and an auditory cue with no associated reward value that elicited no response when presented alone (A2no - depicted as a ringing bell). Their coupling during testing produced no response enhancement (response probability was the same as to the visual cue alone)

FIGURE 2

Apparatus and an exemplar training profile. Top: The orientation and localization task was performed in a 90-cm diameter perimetry apparatus. Stimulus locations spanned the central 180° of space in 15° intervals (only the central 120° was tested here). Each stimulus location contained a complex of two speakers displaced 4 cm from each other, and positioned 4 cm above a complex of three LEDs at 2 cm separations (the 0° location complex was not used in this experiment). From Gingras et al. (2009). Bottom: Exemplar animal’s percent correct approach performance to the stimulus location, to a different location, and its NoGo responses averaged for each training condition (vertical bars show SEM across locations). Approach responses to the A1V1hi and V2low stimuli were both rewarded, and all stimuli were presented at their maximum intensity. Note that A1V1hi and V2lo ultimately elicited similar performance levels despite being associated with different reward magnitudes, and that the animal ultimately opted for a NoGo response to the unrewarded A2no stimulus

FIGURE 3

Reward association determined multisensory enhancement; coupling probability had no effect. Top: Training and testing conditions. Icons are used to illustrate different training and testing conditions. Visual stimuli are depicted as pairs of LEDs with an arrow indicating different directions (left = V1, right = V2) of apparent motion. Auditory stimuli are illustrated with a speaker from which either low-pass sound (from the bottom, A1) or high-pass sound (from the top, A2) are emanating. Reward is illustrated by a food bowl with either 2 (low) or 4 (high) food pellets or an X indicating that no response was ever rewarded in that condition. Animals were first trained to approach the A1V1hi stimulus and the V2low stimulus. The A2no stimulus was presented during training but was not associated with reward. Testing included all four modality-specific stimuli presented individually as well as in every possible cross-modal combination. Bottom: Results of Testing. Shown are the percent of approach responses on unisensory (V1hi, V2low) and multisensory (A1V1hi, A2V1low, A1V2hi, A2V2low) trials. Note that the V1hi and V2low stimuli elicited comparable correct performance levels despite being associated with different reward magnitudes. A1no (paired with reward in training) elicited reliable orientation responses even though it was not individually rewarded during testing. A2no (dissociated from reward in both training and testing) did not. The summary metric MEv (right ordinate) quantifies the proportionate enhancement in the ability to correctly approach each visual target when it is combined with each auditory stimulus. Multisensory conditions involving A1no showed significant multisensory enhancement (leftmost two plots), while conditions involving A2no did not (rightmost two plots). Connected markers indicate individual animal performance for each cue. Error bars on markers indicate SEM across locations. *indicates significance of p < .05, **indicates significance of p < .01

FIGURE 4

Discriminating the visual stimuli and rewards. To determine if they were discriminable, V1hi and V2low were simultaneously presented at homotopic loci (−60°/+60°, −45°/+45°, −30/+30°, −15°/+15°). The animal could respond to either stimulus and receive a reward at the assigned level (see also Dakos et al., 2019). A significant (p = .011) preference was observed for the stimulus associated with the higher reward. Data were normalized by total number of approach responses. As expected, the animal chose the V1hi stimulus more frequently, with a proportionate preference reflecting the proportionate difference in their reward values (2:1)

FIGURE 5

When combined with a familiar visual stimulus, a novel auditory stimulus elicited robust multisensory enhancement. Top: Training and Testing Conditions. Animals began testing on the A3no stimulus and cross-modal configurations directly after completing experiment 1.A1V1hi and A2V2low conditions were included in the tests. Bottom: Results of Testing. The two leftmost plots illustrate conditions replicating the results Experiment 1 (see Figure 3). The two rightmost plots illustrate the results of tests pairing the novel auditory stimulus with each visual stimulus. Both combinations elicit multisensory enhancement at near ceiling levels. Conventions are the same as Figure 3

FIGURE 6

The effect of reversing stimulus identity and reward association. Data from an exemplar animal are shown during training in Experiment 3. Animals immediately performed at ceiling levels. Conventions are the same as Figure 2

FIGURE 7

Reversing reward contingencies reversed the pattern of multisensory enhancement. Top: Training and Testing Conditions. Animals were retrained with covariance presentations and reward outcomes reversed from Exp. 1. Bottom: Results of Testing. The two leftmost plots illustrate the results of testing with combinations involving the A2 stimulus, which now, in the A2V2hi combination yielded significantly enhanced responses. The two rightmost plots illustrate the results of testing with combinations involving the A1 stimulus, which had not been trained in combination with a rewarded visual stimulus since Experiment 1, which elicited significant but marginal levels of enhancement. All together, the pattern of multisensory enhancement is (partially) reversed from what was seen in Exp. 1 (Figure 3), reflecting the reversal of the stimulus associations in training. Conventions are the same as Figure 3