An ecological study protocol for the multimodal investigation of the neurophysiological underpinnings of dyadic joint action

Abstract

A novel multimodal experimental setup and dyadic study protocol were designed to investigate the neurophysiological underpinnings of joint action through the synchronous acquisition of EEG, ECG, EMG, respiration and kinematic data from two individuals engaged in ecologic and naturalistic cooperative and competitive joint actions involving face-to-face real-time and real-space coordinated full body movements. Such studies are still missing because of difficulties encountered in recording reliable neurophysiological signals during gross body movements, in synchronizing multiple devices, and in defining suitable study protocols. The multimodal experimental setup includes the synchronous recording of EEG, ECG, EMG, respiration and kinematic signals of both individuals via two EEG amplifiers and a motion capture system that are synchronized via a single-board microcomputer and custom Python scripts. EEG is recorded using new dry sports electrode caps. The novel study protocol is designed to best exploit the multimodal data acquisitions. Table tennis is the dyadic motor task: it allows naturalistic and face-to-face interpersonal interactions, free in-time and in-space full body movement coordination, cooperative and competitive joint actions, and two task difficulty levels to mimic changing external conditions. Recording conditions-including minimum table tennis rally duration, sampling rate of kinematic data, total duration of neurophysiological recordings-were defined according to the requirements of a multilevel analytical approach including a neural level (hyperbrain functional connectivity, Graph Theoretical measures and Microstate analysis), a cognitive-behavioral level (integrated analysis of neural and kinematic data), and a social level (extending Network Physiology to neurophysiological data recorded from two interacting individuals). Four practical tests for table tennis skills were defined to select the study population, permitting to skill-match the dyad members and to form two groups of higher and lower skilled dyads to explore the influence of skill level on joint action performance. Psychometric instruments are included to assess personality traits and support interpretation of results. Studying joint action with our proposed protocol can advance the understanding of the neurophysiological mechanisms sustaining daily life joint actions and could help defining systems to predict cooperative or competitive behaviors before being overtly expressed, particularly useful in real-life contexts where social behavior is a main feature.

Keywords: dyadic motor task; electroencephalography; joint action; kinematic data; multimodal experimental setup; synchronization; table tennis.

FIGURE 1

Schematic representation of the experimental setup comprising two mobile EEG devices and one stationary motion capture system.

FIGURE 2

Positions of the bipolar EMG sensors on each table tennis player.

FIGURE 3

Design of the novel Flower electrode and cap: (A) cap from the outside showing the equidistant electrode layout (di Fronso et al., 2019) and (B) turned inside out showing the individual flower electrodes; (C) magnification of a single flower electrode with 30 pins (Warsito et al., 2023).

FIGURE 4

Overview of the novel synchronization solution for control, monitoring and post hoc data alignment in multimodal joint action studies.

FIGURE 5

Example of synchronous multimodal physiological data recorded in one dyad during cooperative table tennis. Panels (A–E) refer to one member of the dyad (player 1), whereas panels (F–J) refer to the second member of the dyad (player 2). Panels (A,F) show the raw dry EEG recordings (64 channels, amplitude in μV). Panels (B,G) show the cardiac signals (one channel, amplitude in mV). Panels (C,H) show the respiration effort (one channel, amplitude in%). Panels (D,I) show the electromyographic signals recorded from four muscles per player (four channels, amplitude in mV). Panels (E,J) show the absolute value of the position vector of the marker placed on top of the table tennis racket (one trace, amplitude in m). The time scale at the bottom of the figure is identical for all panels and both players. The vertical red and green lines over all panels indicate the start and stop event triggers used for synchronization purposes.

FIGURE 6

Schematic representation of the masks that have been tested to identify the ideal hard difficulty level. (A) Area reduction with a circle mask. (B) Area reduction with a square mask. The area of the table is reduced by 10, 15, 20, 25, and 30%, starting from the smallest internal mask to the largest external one in both pictures. The players can use only the blue area of the table tennis table.

FIGURE 7

Analysis of the reported Borg scale values. The horizontal red lines indicate the mean Borg scale values for the easy difficulty level. The height of the columns indicates the mean Borg scale value for the hard difficulty level, and the error bars indicate the Standard Error Mean (SEM). Significant differences between the easy and the hard difficulty levels are marked (*p < 0.05 and **p < 0.005). (A) Results obtained during cooperation. (B) Results obtained during competition.

FIGURE 8

Mean rally duration. The red lines indicate the mean rally duration for the easy difficulty level. The height of the columns indicates the mean rally duration for the hard difficulty level, and the error bars indicate the Standard Error Mean (SEM). Significant differences between the easy and the hard difficulty levels are marked (*p < 0.05). (A) Results obtained during cooperation. (B) Results obtained during competition.

FIGURE 9

Examples of hyperbrain functional connectivity maps during the performance of cooperative and competitive table tennis. The maps are obtained from one dyad and from subsets of 19 channels representative of all brain areas (one subset for each member of the dyad). Each panel (A– F) refers to a unique combination of condition (cooperation or competition) and frequency band in which the EEG data were analyzed (theta, alpha, and beta bands). Each panel includes two rows, each referring to one of the two dominant hyperbrain functional connectivity maps observable for that specific combination of condition and frequency band. In each row the functional connectivity maps are shown, from left to right, as phase synchronization maps [calculated with the Phase Lag Index (PLI)], as adjacency matrices (where only the 30% of the highest functional connections are retained), and as dominant interbrain functional connections. At the right-hand side of the figure, there are the color scales for the maps: the one at the top refers to the PLI maps, whereas the one at the bottom refers to the adjacency matrices.

FIGURE 10

Examples of vectors expressing the position of the coated ball during cooperative and competitive table tennis [(A,B) panels, respectively]. The X-axis expresses the time in seconds, whereas the Y-axis expresses the ball position in meters. The gray and orange rectangles indicate the table tennis rallies during cooperation and competition, respectively.

FIGURE 11

Representation of the targets positioned on one half of the table tennis table to test the service skill.

FIGURE 12

Structure of the study paradigm. Each rectangle in the bar represents an individual data acquisition session of 2.5 min duration. The cooperation and competition sessions played at easy and hard difficulty levels are executed in randomized order (not indicated in the figure).