Exploring action dynamics as an index of paired-associate learning

Abstract

Much evidence exists supporting a richer interaction between cognition and action than commonly assumed. Such findings demonstrate that short-timescale processes, such as motor execution, may relate in systematic ways to longer-timescale cognitive processes, such as learning. We further substantiate one direction of this interaction: the flow of cognition into action systems. Two experiments explored match-to-sample paired-associate learning, in which participants learned randomized pairs of unfamiliar symbols. During the experiments, their hand movements were continuously tracked using the Nintendo Wiimote. Across learning, participant arm movements are initiated and completed more quickly, exhibit lower fluctuation, and exert more perturbation on the Wiimote during the button press. A second experiment demonstrated that action dynamics index novel learning scenarios, and not simply acclimatization to the Wiimote interface. Results support a graded and systematic covariation between cognition and action, and recommend ways in which this theoretical perspective may contribute to applied learning contexts.

Figure 1. The experimental display and interface.

Top panel: The Wiimote is held in the dominant hand, with the thumb engaging the remote's A button to click in the experimental software. Middle panel: A view of the overall context, with the light in the room on. The arm is held above the projector, and the remote controls a cursor that selects the correct match. In this image, the trial initiation target at the bottom center is present. Bottom panel: Participants performed the learning task with the lights dimmed.

Figure 2. Visualization of extractable Wiimote data.

Left panel: An example Wiimote trajectory from the bottom center to the right match. The trajectory demonstrates the sway present in the arm prior to committing to the movement, during latency. When the arm moves outside the 100-pixel escape region (dotted line), this provides the in-motion segment of the trajectory. The small dotted square is the location at which a button press is initiated. Right panel: The x- (green) and y-axis (blue) accelerometer data for that same trial. During latency, a resting voltage signal is present (the at-rest voltage generated by the accelerometers). This voltage is modulated (up or down, depending on direction of movement) when the Wiimote is displaced, shown between the two dotted boxes. This example trial has a brief in-motion segment. Acceleration range was computed by subtracting the mean latency acceleration from the absolute maximum acceleration during motion. The rightmost dotted box is the region in which perturbation of the remote is inspected for voltage range.

Figure 3. Results of Experiment 1, with means computed over presentation order for each symbol pair.

All cursor measures diminish across symbol presentation order, while acceleration measures rise.

Figure 4. Presentation of the interaction between correct (green line) and incorrect (red points) trials and presentation order.

Latency does not seem to index correct/incorrect trials across presentation order for each pair. However, in-motion time and x-flips drop relatively more for correct trials compared to incorrect trials, indicated by the significant interaction term (see text for details).

Figure 5. Results of Experiment 2 presented by presentation order for each pair.

Each block is presented as a separated line, labeled using block number and shaded from dark (first block) to lighter (third block) lines.

Figure 6. Sample entropy for each presentation order collapsed across blocks.

Values initially rise then drop near final presentation orders.