Audio-visual concert performances synchronize audience's heart rates

Abstract

People enjoy engaging with music. Live music concerts provide an excellent option to investigate real-world music experiences, and at the same time, use neurophysiological synchrony to assess dynamic engagement. In the current study, we assessed engagement in a live concert setting using synchrony of cardiorespiratory measures, comparing inter-subject, stimulus-response, correlation, and phase coherence. As engagement might be enhanced in a concert setting by seeing musicians perform, we presented audiences with audio-only (AO) and audio-visual (AV) piano performances. Only correlation synchrony measures were above chance level. In comparing time-averaged synchrony across conditions, AV performances evoked a higher inter-subject correlation of heart rate (ISC-HR). However, synchrony averaged across music pieces did not correspond to self-reported engagement. On the other hand, time-resolved analyses show that synchronized deceleration-acceleration heart rate (HR) patterns, typical of an "orienting response" (an index of directed attention), occurred within music pieces at salient events of section boundaries. That is, seeing musicians perform heightened audience engagement at structurally important moments in Western classical music. Overall, we could show that multisensory information shapes dynamic engagement. By comparing different synchrony measures, we further highlight the advantages of time series analysis, specifically ISC-HR, as a robust measure of holistic musical listening experiences in naturalistic concert settings.

Keywords: cardiorespiratory synchrony; engagement; inter‐subject correlation; music concerts; stimulus−response correlation.

FIGURE 1

Outline of the experimental design and analysis pipelines. (A) shows the study design: audiences were presented with music pieces in audio‐visual (AV, orange box) and audio‐only (AO, blue box) conditions, while heart (ECG), respiration, and acoustic signal of the music (maroon) were continuously recorded. (B) shows how beats per minute (BPM) were extracted from the ECG and respiration signals, which were then used for correlational measures (upper subpanel). (B) also shows that the angles of heartbeat and respiration cycles were extracted for phase coherence measures, where the start of the cycle began at the peaks of the ECG and respiration signals (peaks marked as green dots) (lower subpanel). (C) shows the correlational measures (upper subpanel) and the phase‐coherence measures (lower subpanel) that we extracted from the continuous measures. For the correlational measures, we extracted the heart and respiration rates (HR, RR, from cardiorespiratory signals) and the spectral flux (from acoustic signal). For the phase coherence measures, we extracted the phase angle of the ECG and respiration cycles (from cardiorespiratory signals), and the angles of the spectral flux band‐pass filtered at the audience's heart frequencies (0.951–1.159 Hz) and respiration frequencies (0.268–0.358 Hz). (D) shows the extracted synchrony measures: stimulus–subject (vertical, left) and inter‐subject (vertical, right) for correlation (horizontal, top) and phase coherence (horizontal, bottom) measures. For each of the stimulus–subject measures, we applied a constant lag for all participant physiological responses to account for the time taken to respond to the music.

FIGURE 2

Raincloud plots showing modality differences of audio‐only (blue) and audio‐visual (orange) conditions for time‐averaged synchrony measures of SRC‐HR (far left), ISC‐HR (left), and ISC‐RR (right) as well as self‐reported engagement (far right). Smaller points represent each data point, while the larger point represents the mean; error bars represent standard error of the mean. The shape of the distribution is shown by the half violin.

FIGURE 3

Synchrony values at 15 s prior to and after section boundaries for stimulus correlation of HR (left), inter‐subject correlation of HR (middle), and inter‐subject correlation of RR (right). The zero point represents the section boundary. Thick lines represent the average, and the ribbon represents the standard error of the mean for audio‐only (AO, blue) and audio‐visual (AV, orange) conditions. * represents a significant modality effect within that time window.

FIGURE 4

Heart rate (HR, left) and respiration rate (RR, right) BPM values at 15 s prior to and after section boundaries. The zero point represents the section boundary. Thick lines denote the average BPM, while ribbons around the average line represent the standard error.