Principles of multisensory behavior

Abstract

The combined use of multisensory signals is often beneficial. Based on neuronal recordings in the superior colliculus of cats, three basic rules were formulated to describe the effectiveness of multisensory signals: the enhancement of neuronal responses to multisensory compared with unisensory signals is largest when signals occur at the same location ("spatial rule"), when signals are presented at the same time ("temporal rule"), and when signals are rather weak ("principle of inverse effectiveness"). These rules are also considered with respect to multisensory benefits as observed with behavioral measures, but do they capture these benefits best? To uncover the principles that rule benefits in multisensory behavior, we here investigated the classical redundant signal effect (RSE; i.e., the speedup of response times in multisensory compared with unisensory conditions) in humans. Based on theoretical considerations using probability summation, we derived two alternative principles to explain the effect. First, the "principle of congruent effectiveness" states that the benefit in multisensory behavior (here the speedup of response times) is largest when behavioral performance in corresponding unisensory conditions is similar. Second, the "variability rule" states that the benefit is largest when performance in corresponding unisensory conditions is unreliable. We then tested these predictions in two experiments, in which we manipulated the relative onset and the physical strength of distinct audiovisual signals. Our results, which are based on a systematic analysis of response time distributions, show that the RSE follows these principles very well, thereby providing compelling evidence in favor of probability summation as the underlying combination rule.

Figure 1.

The redundant signal paradigm and the probability summation framework. a, Participants indicated the onset of two target signals (motion or sound; see b) that were embedded in a continuous audiovisual background. Except for similar stimulus onset, these distinct signals did not refer to a common environmental property. b, We presented signals in three basic conditions according to the classical redundant signal paradigm. In two single signal conditions, either the motion (M) or the sound (S) signal was presented. In a redundant signal condition (R), both signals were presented. Signals are “redundant” in the sense that detection of either signal was sufficient for a correct response. Hence, the two signals are coupled by a logical OR operator by the definition of the task. c, According to the LATER model of perceptual decision making, on presentation of M, evidence for the motion signal is accumulated from the start level L₀ until the threshold L_T is reached and a response is triggered. The drift rate is subject to Gaussian noise (see D_M; the drift rate of an exemplary trial is indicated by the thick diagonal line). The resulting RT distribution follows the reciprocal of the Gaussian, which is a reci-normal distribution that is skewed to the right like empirical distributions. d, Evidence for the sound signal is accumulated analogously. e, On presentation of R, according to race models, a response on a given trial can be triggered by the faster of the two parallel, stochastic decision processes (in the illustrated exemplary trial, when evidence for sound reaches L_T). Consequently, compared with the single signal conditions, a speedup of RTs is expected on average across trials. Within the probability summation framework (for more details, see Materials and Methods), the resulting RT distribution can be computed by the minimum function of the RT distributions shown in c and d (which corresponds to the maximum function of the corresponding drift rate distributions).

Figure 2.

Numerical analysis of the principle of congruent effectiveness. We assume that RTs to a given signal X follow a fixed cumulative distribution P_X. To maximize the benefit, signal Y can be manipulated freely so that P_Y can take the form of any cumulative distribution. a, According to Equation 5 (see Materials and Methods), the benefit for a fixed value of P_X, here for example equal to 0.3, is given by the minimum function of the two linear functions f₁ = P_Y(1 − P_X) and f₂ = P_X(1 − P_Y). This minimum function is strictly increasing with P_Y when f₁ < f₂ and is strictly decreasing with P_Y when f₁ > f₂. It follows that the minimum function has an absolute maximum when f₁ = f₂, that is, when P_Y is equal to 0.3 as well. b, The same is true for any value of P_X between 0 and 1, that is, the benefit is always largest when P_Y is equal to P_X (the dotted line indicates the cross-section shown in a). It follows that the expected benefit using a fixed signal X is largest if signal Y is chosen so that RTs in the corresponding single conditions are equal in distribution. Moreover, this analysis shows that the principle of congruent effectiveness does not depend on distributional assumptions (as made, for example, by the model presented in Fig. 1).

Figure 3.

Benefits expected from probability summation. We simulated the RSE for arbitrary signals X and Y using the model illustrated in Figure 1c--e (under the assumption of statistical independence). In each plot, RTs to signal X follow a reci-normal distribution with a fixed median and MAD (as indicated by vertical and horizontal lines, respectively). RTs to signal Y could follow any reci-normal distribution within the plotted range of median RTs and MADs. We then computed the expected benefit for all combinations of signals X and Y (Eq. 5). These computations reveal two basic properties to predict benefits within the probability summation framework. First, for each plot, the maximal benefit is observed when RTs to signals X and Y had identical medians as well as identical MADs (or, more precisely, when RTs to signal X and Y are equal in distribution). We refer to this property as the principle of congruent effectiveness. Second, when we varied independently the median and the MAD of RTs to signal X across plots, benefits are mostly enhanced with increasing MADs (middle column). In contrast, manipulations of the median had basically no effect (middle row). We refer to this property as the variability rule.

Figure 4.

The RSE for different values of SOA (experiment 1). a, Median RTs as a function of SOA (mean across 40 blocks of 40 trials each; error bars indicate the SEM). RTs were fastest for redundant signals presented with an SOA of 0 ms. Critically, the speedup compared with the corresponding single signal conditions was largest when the SOA was 30 ms (broken lines indicate projected median RTs for single signals for different values of SOA). b, Benefits as a function of the difference in median RT in single conditions (negative values indicate that median RTs to sound compared with motion signals were faster taking into account the delays resulting from the SOA manipulation; labeling of SOA conditions as in d). As expected from probability summation, benefits were largest when the difference in median RTs approached 0 ms. Estimated benefits were based on cumulative group RT distributions with 1600 trials per distribution. Vertical lines show 95% confidence intervals, calculated by 1000 repetitions of a bootstrap procedure. c, Cumulative group RT distributions in the condition with an SOA of 0 ms. For each distribution, symbols indicate the empirical quantiles (each quantile is based on 40 trials). In the single conditions, the dotted lines indicate best fitting reci-normal distributions (see Table 1). Based on these distributions, we fitted the probability summation model with the correlation ρ and the additional noise η as free parameters to the empirical distribution with redundant signals (Fit). The model fitted the empirical distribution very well (for best-fitting model parameters, see Table 2). M, Motion; S, sound; R, redundant. d, Benefits (see b) as a function of predictions based on the model fitted in the redundant condition with an SOA of 0 ms (see c). Interestingly, the model predicted the benefits for the remaining SOA conditions reasonably well.

Figure 5.

RT distributions (experiment 2). a–i, Each plot shows the cumulative group RT distributions for the combination of motion and sound signals at different levels of signal strength [bottom to top rows: weak (W), medium (M), and strong (S) sound signals; left to right columns: weak, medium, and strong motion signals]. For single signal conditions (Motion and Sound), broken lines indicate the best-fitting reci-normal distributions (for best-fitting parameters, see Table 3; empirical quantiles are not shown for clarity). For redundant signal conditions, dots indicate the empirical quantiles of the cumulative group RT distribution (each of the 50 quantiles is based on 40 trials, i.e., each distribution is based on 2000 trials). We fitted the probability summation model, which is constrained by the single signal conditions and has two degrees of freedom (i.e., the correlation ρ and the additional noise η), to the empirical quantiles in the redundant condition with medium signals (see e). The model fitted the data virtually perfectly (as indicated by the solid dark gray line; Fit). Interestingly, by keeping identical parameters, the model predicted the distributions of the remaining conditions, except for the condition with a pair of weak signals (see g), surprisingly well (as indicated by the solid light gray lines; Prediction). For the subsequent analysis shown in Figure 6, we computed the benefit for each pair of signals based on the area between the CDF with redundant signals and the faster of the CDFs with single signals (as illustrated in c; see also Eq. 3).

Figure 6.

Benefits for different combinations of signal strength (experiment 2). a, Benefits as a function of signal strength of motion and sound signals (w, weak; m, medium; s, strong signals). The arrows highlight the conditions with pairs of weak, medium, and strong signals in both modalities. b, Benefits as a function of the parameter free predictions based on probability summation under the assumption of statistical independence (Eq. 5). c, Benefits as a function of predictions based on the model fitted in the redundant condition with medium signals (see Fig. 5e). Benefits in each condition were estimated based on the cumulative group RT distributions with 2000 trials per distribution (as illustrated in Fig. 5c). Vertical lines show 95% confidence intervals, calculated by 1000 repetitions of a bootstrap procedure.

Figure 7.

Additional noise. For each of the 14 tested redundant signal conditions, we show the best-fitting estimates of the additional noise parameter η as a function of the median difference in the corresponding single signal conditions. Estimates of the noise were based on the model fitting with cumulative group RT distributions with 1600 and 2000 trials per distribution in experiments 1 and 2, respectively (see Tables 2, 4). Negative values of median difference indicate that median RTs to sound signals were faster compared with motion signals.