Multisensory perceptual learning reshapes both fast and slow mechanisms of crossmodal processing

Abstract

Previous research has shown that sounds facilitate perception of visual patterns appearing immediately after the sound but impair perception of patterns appearing after some delay. Here we examined the spatial gradient of the fast crossmodal facilitation effect and the slow inhibition effect in order to test whether they reflect separate mechanisms. We found that crossmodal facilitation is only observed at visual field locations overlapping with the sound, whereas crossmodal inhibition affects the whole hemifield. Furthermore, we tested whether multisensory perceptual learning with misaligned audio-visual stimuli reshapes crossmodal facilitation and inhibition. We found that training shifts crossmodal facilitation towards the trained location without changing its range. By contrast, training narrows the range of inhibition without shifting its position. Our results suggest that crossmodal facilitation and inhibition reflect separate mechanisms that can both be reshaped by multisensory experience even in adult humans. Multisensory links seem to be more plastic than previously thought.

Fig. 1

Setup, design, and hypothesis. a) Setup. Sounds appeared to the left or right (indicated by dashed speakers). Visual field locations (dotted circles) were aligned (A), proximal (P) or distal (D) to the sound source on either side. b) Design. During test sessions (pre and post), crossmodal interactions were assessed for all locations (D_t/P_t/A/P_u/D_u) and different sound-Gabor delays (SOA: 150/300/1,000 ms). During the eight training sessions, a sound was paired with a Gabor at P_t, A, or P_u (left or right). Subsequently, a square or circle encompassed the Gabor. Target shapes established a contingency between same-side sounds and Gabors at trained locations (P_t) only. The distal locations next to P_t (D_t) or next to P_u (D_u) were not stimulated. c) Crossmodal facilitation at the generic locations (A) prior to training was expected to shift to the trained locations (P_t) after training (only one side depicted)

Fig. 2

Test results prior to training. a) Percentage correct Gabor discrimination. For the 150-ms stimulus onset asynchrony (SOA), Gabors were discriminated more correctly when preceded by valid (V) than by invalid (I) sound cues. This crossmodal facilitation was observed only at aligned (A) but not at proximal (P) or distal (D) locations. No significant (n.s.) differences emerged for the 300-ms SOA. For the 1,000-ms SOA, crossmodal inhibition was observed across all visual field locations (A, P, D). Dashed lines depict the mean across all aperture locations. See ANOVA results in text. b) No significant differences were observed on response times. However, response time differences tended to complement validity effects on percentage correct. Upper and lower visual field representations of proximal (P) and distal (D) apertures were averaged for illustrative purposes. They were separately analyzed in statistical tests (see text). Error bars reflect within-subjects SEM (Loftus & Masson, 1994) for the factor Validity. n = 12. * p ≤ .05. † p ≤ .10

Fig. 3

Validity effects (VE). a) Pre- and post-training tests. Prior to training, a VE (performance at same/valid side as sound minus performance at opposite/invalid side) showing crossmodal facilitation emerged only at the aligned (A) location for the 150-ms stimulus onset asynchrony (SOA). Crossmodal inhibition was observed at the 1,000-ms SOA with no significant differences across vertical locations. Error bars reflect within-subjects SEM (Loftus & Masson, 1994) for the VE. b) Change in VE (post-training minus pre-training). Crossmodal facilitation decreased at the aligned (A) and increased at the trained location (P_t). Crossmodal inhibition decreased at distal (D_t/D_u) locations. No significant change was observed at aligned or proximal locations (P_t/A/P_u). Dashed lines depict the mean for both distal (D_t/D_u) locations and the mean for aligned or proximal locations (P_t/A/P_u). Error bars reflect within-subjects SEM (Loftus & Masson, 1994) for the difference in the VE. n = 12. * p ≤ .05

Fig. 4

Shape detection during training. a) Sensitivity measures (d′). b) Mean response times. Shape detection improved at all visual field locations (P_t, A, P_u) and both when sounds were presented at the same (valid) side (V) or the opposite (invalid) side (I). In addition, a location-specific change of the validity effect was observed on response times (see Fig. 5). Two consecutive training sessions were pooled. Error bars reflect within-subjects s.e.m. (Loftus & Masson, 1994) for the factor Training Session. n = 17

Fig. 5

Change of validity effect during training. During early training, crossmodal facilitation (VE = response times at opposite/invalid side as sound minus response times at same/valid side) only emerged at aligned (A) locations. At later sessions, crossmodal facilitation was restricted to trained locations (P_t). Error bars reflect within-subjects SEM (Loftus & Masson, 1994) for the VE. n = 17. * p ≤ .05