Learning to associate auditory and visual stimuli: behavioral and neural mechanisms

Abstract

The ability to effectively combine sensory inputs across modalities is vital for acquiring a unified percept of events. For example, watching a hammer hit a nail while simultaneously identifying the sound as originating from the event requires the ability to identify spatio-temporal congruencies and statistical regularities. In this study, we applied a reaction time and hazard function measure known as capacity (e.g., Townsend and AshbyCognitive Theory 200-239, 1978) to quantify the extent to which observers learn paired associations between simple auditory and visual patterns in a model theoretic manner. As expected, results showed that learning was associated with an increase in accuracy, but more significantly, an increase in capacity. The aim of this study was to associate capacity measures of multisensory learning, with neural based measures, namely mean global field power (GFP). We observed a co-variation between an increase in capacity, and a decrease in GFP amplitude as learning occurred. This suggests that capacity constitutes a reliable behavioral index of efficient energy expenditure in the neural domain.

Figure 1

Gabor patches oriented diagonally. The visual stimuli, when present consisted of 1 of 6 levels of frequency as shown in this figure. From the top left to the bottom right cycles in Hz: 4 Hz, 5 Hz, 6.6 Hz, 8.3 Hz, 12.5 Hz, and 25 Hz. Auditory pure tones were created at 400 Hz, 500 Hz, 660 Hz, 830 Hz, 1250 Hz, and 2500 Hz.

Figure 2

The left panel (A) shows d’ averaged across observers, as a function of learning. The results strongly indicate an overall increase in discriminability across days, with the largest increase occurring between days 1 and 2. The error bars indicate 1 standard error of the mean. The panel on the right (B) shows c (criteria) as a function of learning. Interestingly, the results fail to reveal a pattern between training day and response criteria.

Figure 3

This figure displays the log integrated hazard functions, ln[-ln[S(t)]] or ln[H(t)], across each training day for each of the four observers. As one may observe, the functions are proportional to one another, and for participants 2-4, increase as learning occurs.

Figure 4

Each panel plots the mean GFP separately for each training day. The panels in the column on the left show the mean GFP for the audiovisual matched trials, and the panels on the right show the mean GFP for the mismatched trials. The top row shows participant 1’s GFP data, the second row participant 2’s, the third row participant 3’s, and the fourth row, participant 4’s.

Figure 5

This bar graph shows mean GFP as a function of training day for each participant (rows 1-4; see Figure 4). The panels in the left column show mean GFP averaged over the interval ranging from 250-350 ms post stimulus onset, while the panels in the right column show the mean GFP averaged over the 0-100 ms interval.