Audiovisual Modulation in Music Perception for Musicians and Non-musicians

Abstract

In audiovisual music perception, visual information from a musical instrument being played is available prior to the onset of the corresponding musical sound and consequently allows a perceiver to form a prediction about the upcoming audio music. This prediction in audiovisual music perception, compared to auditory music perception, leads to lower N1 and P2 amplitudes and latencies. Although previous research suggests that audiovisual experience, such as previous musical experience may enhance this prediction, a remaining question is to what extent musical experience modifies N1 and P2 amplitudes and latencies. Furthermore, corresponding event-related phase modulations quantified as inter-trial phase coherence (ITPC) have not previously been reported for audiovisual music perception. In the current study, audio video recordings of a keyboard key being played were presented to musicians and non-musicians in audio only (AO), video only (VO), and audiovisual (AV) conditions. With predictive movements from playing the keyboard isolated from AV music perception (AV-VO), the current findings demonstrated that, compared to the AO condition, both groups had a similar decrease in N1 amplitude and latency, and P2 amplitude, along with correspondingly lower ITPC values in the delta, theta, and alpha frequency bands. However, while musicians showed lower ITPC values in the beta-band in AV-VO compared to the AO, non-musicians did not show this pattern. Findings indicate that AV perception may be broadly correlated with auditory perception, and differences between musicians and non-musicians further indicate musical experience to be a specific factor influencing AV perception. Predicting an upcoming sound in AV music perception may involve visual predictory processes, as well as beta-band oscillations, which may be influenced by years of musical training. This study highlights possible interconnectivity in AV perception as well as potential modulation with experience.

Keywords: audiovisual; auditory; event-related potential (ERP); inter-trial phase coherence (ITPC); music perception; musical experience; musicians and non-musicians.

Figure 1

Timeliness for the audio only, video only, and audiovisual stimuli.

Figure 2

Grand averaged event-related potentials at Cz and topographical maps for N1 and P2, plotted for audio only (blue) and audiovisual minus video only (red).

Figure 3

Musicians and non-musicians' ITPC spectrum (0.5–48 HZ) for audio only (AO) and audiovisual minus video only (AV-VO).

Figure 4

Pointwise two-tailed t-test comparison of AO and AV-VO ERPs at F3, Fz, F4, C3, C4, P3, Pz, P4.