Phasic and sustained interactions of multisensory interplay and temporal expectation

Abstract

Every moment organisms are confronted with complex streams of information which they use to generate a reliable mental model of the world. There is converging evidence for several optimization mechanisms instrumental in integrating (or segregating) incoming information; among them are multisensory interplay (MSI) and temporal expectation (TE). Both mechanisms can account for enhanced perceptual sensitivity and are well studied in isolation; how these two mechanisms interact is currently less well-known. Here, we tested in a series of four psychophysical experiments for TE effects in uni- and multisensory contexts with different levels of modality-related and spatial uncertainty. We found that TE enhanced perceptual sensitivity for the multisensory relative to the best unisensory condition (i.e. multisensory facilitation according to the max-criterion). In the latter TE effects even vanished if stimulus-related spatial uncertainty was increased. Accordingly, computational modelling indicated that TE, modality-related and spatial uncertainty predict multisensory facilitation. Finally, the analysis of stimulus history revealed that matching expectation at trial n-1 selectively improves multisensory performance irrespective of stimulus-related uncertainty. Together, our results indicate that benefits of multisensory stimulation are enhanced by TE especially in noisy environments, which allows for more robust information extraction to boost performance on both short and sustained time ranges.

Figure 1

Experimental Design. (A) Three exemplary stimulus sequences (left: auditory-only, middle: multisensory with redundant target, right: multisensory with just auditory target). Each trial started with a blank screen (inter-trial-interval) followed by a sequence of 11 auditory (Exp 1 + 2), visual (Exp 1 + 2), or audiovisual stimuli (all Exp). Stimuli were presented for 100 ms with a 100 ms gap in between. The stimulus sequence ended with a blank screen (response window, max. 1500 ms). Targets were either presented at the 3rd or 9th position. Note that squares highlight the target (lower or higher frequency than distractor items) for illustrative purposes only and were not present in the experiment. In Experiments 3 and 4 only multisensory sequences were used (middle and left sequence). (B) Overview of factorial experimental design. Depicted are all 6 possible target conditions and factor levels of spatial and target uncertainty. The experiment’s number (bottom) denotes in which experiment a particular combination of factor levels was used. Note that manipulations of temporal expectations were identical across experiments.

Figure 2

Inferential and modelling results. (A) Mean dʹ values (+/−SE) of the significant interactions “modality × TE” and “modality × TE × spatial uncertainty”. dʹ values are displayed for the audiovisual (AV) vs. best unisensory conditions (best[A,V]). Expected trials are displayed in dark grey, unexpected trials in light grey. (B) Mean dʹ values for sequential effect (n – 1) interactions of “modality × TP”. Target position (TP) match trials are displayed in dark grey, TP mismatch trials in light grey. (C) Results of the d′ modelling analysis. Left: Best model prediction of individual data points (x). The black line depicts a perfect prediction and the red dotted line the regression of all data points. Right: Interaction of MSI with unisensory preference. Red dotted line is regression of all data points. Y-values above the black indicate multisensory facilitation (AV > best[A,V]). Large X-values indicate that participants strongly preferred one modality (max[A,V]≫min[A,V]).