Flexible Visuomotor Associations in Touchscreen Control

Abstract

To move real objects, our hand needs to get in direct physical contact with the object. However, this is not necessarily the case when interacting with virtual objects, for example when displacing objects on tablets by swipe movements. Here, we performed two experiments to study the behavioral strategies of these movements, examining how visual information about the virtual object is mapped into a swipe that moves the object into a goal location. In the first experiment, we investigated how swiping behavior depends on whether objects were located within or outside the swiping workspace. Results show that participants do not start the swipe movement by placing their finger on the virtual object, as they do when reaching to real objects, but rather keep a systematic distance between the object location and the initial swipe location. This mismatch, which was experimentally imposed by placing the object outside the workspace, also occurred when the object was within the workspace. In the second experiment, we investigated which factors determine this mismatch by systematically manipulating the initial hand location, the location of the object and the location of the goal. Dimensionality reduction of the data showed that three factors are taken into account when participants choose the initial swipe location: the expected total movement distance, the distance between their finger on the screen and the object, and a preference not to cover the object. The weight given to each factor differed among individuals. These results delineate, for the first time, the flexibility of visuomotor associations in the virtual world.

Keywords: principal component analysis (PCA); reaching; spatial mismatch touchscreen control; swiping; visuomotor associations.

Figure 1

Stimulus display of Experiment 1 (left panel) and Experiment 2 (right panel). The touchscreen was positioned in front of the participant, centered along the body midline. In both experiments, the red square indicates the initial hand location of the right index finger (start), the white dot indicates the object and the green dot indicates the position (goal) where the participant had to bring the object using a swipe movement. In Experiment 1 (left panel), objects could appear in four directions (right, far, left, near from participant’s body) and two distances (small, large) from the center. The dotted line corresponds to the limits of the workspace during the Instructed Tablet task. Experiment 2 (right panel) consisted only of the Free task with the difference from Experiment 1 that the start, object and goal were arranged along the same horizontal line. Absolute and relative positions of start, object and goal could be varied independently.

Figure 2

(A) Object location and initial swipe location for the Free task (left panel), the Instructed Tablet task (central panel) and the Instructed Reach task (right panel) in Experiment 1. The open circles represent the objects. The dots represent the initial swipe locations averaged across participants. Small striped and larger filled dots show initial swipe locations during trials with small and large goal sizes, respectively. The colors of the circles and the dots represent the object distance from the center (blue, small distance; red, large distance). The black dotted line indicates the line spanned by the start and the object and the black line indicates the spatial mismatch calculated as the difference between the location of the object and the initial swipe location along the dotted line. (B) Object location and initial swipe location in each participant for the Free task in Experiment 1.

Figure 3

Spatial mismatch during Free task (A) and the Instructed Tablet task (B) for each object direction in Experiment 1. Black and gray histograms show the spatial mismatch for objects at small and large distances, respectively; striped and solid histograms show the mismatch during trials with small and large goal sizes. Error bars show the Standard Error.

Figure 4

Object location and initial swipe location for the seven objects during Experiment 2. Each color corresponds to a specific start position, and each symbol correspond to a specific goal position, specified in the schematic legend above the figure.

Figure 5

Spatial mismatch in Experiment 2, (A) as a function of object location on the screen in conditions where the start and goal were in the same location, (B) as a function of the object location relative to the start location (C) and relative to the goal location. Error bars show the Standard Error.

Figure 6

3D slopes of all individual participants in Experiment 2. (A) The three slopes plotted separately. The same participant order is used for all three slopes. The participants with divergent slopes are marked in red. (B) Slopes plotted in a 3D plot. The green grid represents the plane found by applying principal component analysis (PCA) to the 3D slopes. Orthogonal lines connect each point to the plane, to better indicate the 3D positions of the data points and to visualize how close the data are to the plane. The blue line shows the prediction of a preference not obscure the object by the finger. The green dot indicates the prediction of minimizing the expected total movement distance.