Body sway reflects joint emotional expression in music ensemble performance

Abstract

Joint action is essential in daily life, as humans often must coordinate with others to accomplish shared goals. Previous studies have mainly focused on sensorimotor aspects of joint action, with measurements reflecting event-to-event precision of interpersonal sensorimotor coordination (e.g., tapping). However, while emotional factors are often closely tied to joint actions, they are rarely studied, as event-to-event measurements are insufficient to capture higher-order aspects of joint action such as emotional expression. To quantify joint emotional expression, we used motion capture to simultaneously measure the body sway of each musician in a trio (piano, violin, cello) during performances. Excerpts were performed with or without emotional expression. Granger causality was used to analyze body sway movement time series amongst musicians, which reflects information flow. Results showed that the total Granger-coupling of body sway in the ensemble was higher when performing pieces with emotional expression than without. Granger-coupling further correlated with the emotional intensity as rated by both the ensemble members themselves and by musician judges, based on the audio recordings alone. Together, our findings suggest that Granger-coupling of co-actors' body sways reflects joint emotional expression in a music ensemble, and thus provide a novel approach to studying joint emotional expression.

Figure 1

Illustrations of the experimental design and body sway coupling analyses. (a) The retroreflective markers were placed on the head and arms of each performer (the green dots connected by the yellow lines) and on the instruments (the brown dots and lines). From the left to the right are the violinist, pianist, and the cellist. The anterior-posterior body sway motion time series for the three performers, based on the markers on their heads, were extracted for subsequent analyses. (b) Granger causality of body sway reflects the magnitude of information flow from one performer to another performer. The average of the six unique Granger causalities is the causal density (CD), which represents the average amount of information flow across all possible pairs. (c) Cross-correlation of body sway reflects the degree of similarity in each pair of performers. The three unique maximum unsigned cross-correlation coefficients on each trial were averaged for an overall measure of similarity.

Figure 2

Expressivity and Emotion modulate Granger-coupling of body sway, but not correlational-coupling of body sway. (a) A two-way mixed-design ANOVA was conducted on the CD values with Emotion (Happy, Sad) and Expressivity (Expressive, Non-expressive) as factors. The results showed higher CD in the Happy than Sad condition, and higher CD in the Expressive than Non-expressive condition. Each grey dot represents the CD for a trial, and each grey line connects the trials with the same music excerpt under different Expressivity conditions. The red and yellow dots represent the mean CD under Expressive and Non-expressive conditions, respectively. The error bar represents the mean ± one standard error. (b) A two-way mixed-design ANOVA was conducted on the mean cross-correlation coefficient with variables as in (a). The format is the same as (a). The results did not show any significant effect. n.s.: non-significant.

Figure 3

Spearman rank correlation between rated degree of performances and performers’ body sway coupling. The scatter plots in the upper panel represent the Granger-coupling (indexed by CD) of body sway, and plots in the lower panel represent the correlational-coupling (indexed by mean cross-correlation coefficient). Each dot or cross represents the performance of a trial. The dots represent the performers’ self-rated results, and the crosses represent the judges’ rated results. Only the rank correlation coefficients (r_s) and the p-values (p) of the significant correlations are shown.

Figure 4

Rated Emotion-expression and Emotion-valence. Two-way mixed-design ANOVAs were conducted on the mean rated Emotion-expression and Emotion-valence with Emotion (Happy, Sad) and Expressivity (Expressive, Non-expressive) as factors. Each grey dot represents the mean rating across raters (performers or external musicians) for each trial, and each grey line connects the trials with the same music excerpt under different Expressivity conditions. The colored dots represent the mean ratings under Expressive and Non-expressive conditions, respectively.