Hand in motion reveals mind in motion

Abstract

Recently, researchers have measured hand movements en route to choices on a screen to understand the dynamics of a broad range of psychological processes. We review this growing body of research and explain how manual action exposes the real-time unfolding of underlying cognitive processing. We describe how simple hand motions may be used to continuously index participants' tentative commitments to different choice alternatives during the evolution of a behavioral response. As such, hand-tracking can provide unusually high-fidelity, real-time motor traces of the mind. These motor traces cast novel theoretical and empirical light onto a wide range of phenomena and serve as a potential bridge between far-reaching areas of psychological science - from language, to high-level cognition and learning, to social cognitive processes.

Keywords: hand; motor; mouse-tracking; real-time; temporal dynamics; time-course.

Figure 1

Hand movements reveal temporally dynamic face-voice integration. Freeman and Ambady (2011a) found that when categorizing a face's sex, the simultaneous processing of a sex-atypical voice led participants’ mouse trajectories to be continuously attracted to the opposite sex-category response before settling into the response consistent with the face's correct sex. Mean mouse trajectories from one study are depicted (aggregated across male and female targets). In this figure, trajectories for all targets were remapped rightward, with the opposite sex-category on the left and the sex-category consistent with the face's sex on the right. A sample male face stimulus is displayed (all male and female face stimuli were somewhat sex-ambiguous). A voice stimulus typical for the face's sex (masculine) is shown on the right (audio waveform depicted in blue), next to the mean trajectory for sex-typical trials. Its atypical (feminine) counterpart is shown on the left, next to the mean trajectory for sex-atypical trials (audio waveform depicted in purple). During an actual trial, a single face was centered at the bottom of the screen while the voice stimulus played. The bar graph shows trajectories’ maximum deviation toward the opposite sex-category from a direct line between trajectories’ start and end points (error bars denote SE of the mean). The continuous-attraction effect shows that ongoing voice-processing results were dynamically integrated with ongoing face-processing results over time. Adapted from Freeman and Ambady (2011a).

Figure 2

Hand movements provide insights into deception. Duran et al. (2010) had participants answer autobiographical questions either truthfully or falsely by clicking a “yes” or “no” response on the screen. Trajectories’ velocity profiles are plotted as a function of time for truthful responses clicking “no” (dashed line with circles), truthful responses clicking “yes” (dashed line with triangles), false responses clicking “no” (solid line with circles), and false responses clicking “yes” (solid line with triangles). Hand trajectories during false responses reached peak velocity later in the movement. This, among other differences between false and truthful responses in movement complexity and curvature toward the opposite response, suggests the presence of a truth bias that pulls processing off course during the production of a deceptive response. Adapted from Duran et al. (2010).

Figure 3

Parallel competition vs. strategic response switching. Farmer et al. (2007a) conducted an additional study where participants were presented with three colored squares and asked to click on the green square. In the baseline condition, the top and bottom squares were red and the rightward square was green, corresponding to a condition where no attraction to another object on the screen would be predicted. In the switch condition, the green square originally appeared at the top-center, with the other two squares being red. Once the cursor exited the leftward “Start” box, however, the green square turned red, and the leftward square that was originally red became green. This simulates a situation where participants initially commit to an incorrect response, and then subsequently switch their commitment to the correct response. In the competition condition, the bottom square was red, the rightward square was green, and the top-center square was a blue-green. Here, the blue-green square served as a distractor. (A) shows average trajectories. In the baseline condition, the average trajectory showed a direct pursuit toward the correct response. In the switch condition, the average trajectory showed a strong initial movement toward the original location of the green square, followed by a sharp redirection toward the ultimate location of the green square. In the competition condition, the average trajectory was smooth and showed graded attraction toward the location of the distractor. (B) shows the distribution of trial-by-trial deviations for trajectories in the competition condition, illustrating unimodality. (C) shows the distribution of trial-by-trial deviations for trajectories pooled across the switch and baseline conditions, illustrating bimodality. The average trajectories and distributions elicited by these conditions show how the hand-tracking methodology can identifiably tap into a variety of cognitive processes (e.g., parallel competition, strategic response switching). Adapted from Farmer et al. (2007a).