Gaze and hand behaviors during haptic abilities testing-An update to multimedia learning theory

Abstract

The Cognitive Theory of Multimedia Learning (CTML) suggests humans learn through visual and auditory sensory channels. Haptics represent a third channel within CTML and a missing component for experiential learning. The objective was to measure visual and haptic behaviors during spatial tasks. The haptic abilities test (HAT) quantifies results in several realms, accuracy, time, and strategy. The HAT was completed under three sensory conditions using sight (S), haptics (H), and sight with haptics (SH). Subjects (n = 22, 13 females (F), 20-28 years) completed the MRT (10.6 ± 5.0, mean ± SD) and were classified as high or low spatial abilities scores with respect to mean MRT: high spatial abilities (HSA) (n = 12, 6F, MRT = 13.7 ± 3.0), and low spatial ability (LSA) groups (n = 10, 7F, MRT = 5.6 ± 2.0). Video recordings gaze and hand behaviors were compared between HSA and LSA groups across HAT conditions. The HSA group spent less time fixating on mirrored objects, an erroneous answer option, of HAT compared to the LSA group (11.0 ± 4.7 vs. 17.8 ± 7.3 s, p = 0.020) in S conditions. In haptic conditions, HSA utilized a hand-object interaction strategy characterized as palpation, significantly less than the LSA group (23.2 ± 16.0 vs. 43.1 ± 21.5 percent, p = 0.022). Before this study, it was unclear whether haptic sensory inputs appended to the mental schema models of the CTML. These data suggest that if spatial abilities are challenged, LSA persons both benefit and utilize strategies beyond the classic CTML framework by using their hands as a third input channel. This data suggest haptic behaviors offer a third type of sensory memory resulting in improved cognitive performance.

Keywords: cognitive load; haptic abilities; learning; spatial abilities; strategies.

FIGURE 1

Panel A—The redrawn MRT used in electronic interfaces. Participants must determine which two of the four objects on the right are a rotated version of the exemplar on the left. Panel B—Example question for the HAT. The exemplar is located on the left followed by the correct match, mirrored incorrect match and incorrect match. Panel C—The head‐free eye‐tracker with an outfacing scene camera (indicated by red circle) at bridge of nose to collect field of view information, an inward (eye) facing camera and infrared (IR) light on adjustable arms, to record eye movements. Panel D—Gaze‐tracking software recording of subject's pupil under IR light and tracking target. Panel E—Eye tracking scene camera field of view recording of the HAT with pupillary position (indicated by red circle with green outline). Gaze tracking was measured in S (sighted) and SH (sighted with haptics) conditions of the HAT. Panels A and B are used with permission.

FIGURE 2

Static image from video recordings used to determine hand–object interactions. In this image, the subject is reaching through the curtain to interact with HAT objects in the H condition.

FIGURE 3

Participant's visual attention during the HAT represented by gaze fixations during sighted S and SH conditions in HSA and LSA individuals (n = 20). Data are represented as per question means ±1SD. Top Panel A: Average fixation number (FN) indicating how often participant's eyes remain stationary per question. Middle Panel B: Average fixation duration (AFD) indicates how long the eyes remained stationary during each question. Bottom Panel C: Average fixation rate (FR) indicates how often individuals moved their eyes from location to location within their field of view. No significant differences between groups or conditions were found. HAT, haptic abilities test; HSA, high spatial abilities, LSA, low spatial abilities; n, frequency; s, seconds; S, sighted; SD, standard deviation; SH, sighted haptic.

FIGURE 4

Gaze fixation location represented as percent of time to response on each of the 3D objects in the HAT conditions using vision. Each panel illustrates times of HSA and LSA participants (n = 22). Top Panel—The average percent of time fixating on each of the four HAT objects in the S condition. The correct match object was attended to more than the incorrect or mirrored match (p < 0.001). The mirrored match held the attention longer for LSA versus HSA individual (p = 0.020). Bottom Panel—Time fixating on each of the four HAT objects in the SH condition in HSA and LSA individuals (n = 22). The exemplar and correct matched were significantly more attended to than the incorrect and mirrored match (*p < 0.001). Error bars indicated ±1SD. 3D, three‐dimensional; HAT, haptic abilities test; HSA, high spatial abilities, LSA, low spatial abilities; n, frequency; p, p‐value; S, sighted; SD, standard deviation; SH, sighted haptic; %, percent.

FIGURE 5

Mean percent change in pupillary diameter (∆PØ) from participants' baseline diameter in the S, SH, and H sensory conditions of the HAT for HSA and LSA groups (n = 22). No differences between SA groups are present but the main effects of sensory conditions (p = 0.049) are present. Error bars indicate ±1SD. H, haptic; HAT, haptic abilities test; HSA, high spatial abilities, LSA, low spatial abilities; n, frequency; p, p‐value; S, sighted; SD, standard deviation; SH, sighted haptic; %, percent.

FIGURE 6

Percent of object manipulation time per question for each of the four HAT objects in the SH (pattern bars) and H condition (filled bars) using dominant (D) and non‐dominant (ND) hands. No significant differences between the HSA and LSA (n = 21) were found. However, across sensory modalities (SH‐H) and within objects (Exemplar, Correct, Incorrect, Mirrored), objects were manipulated significantly less in the SH condition than in the H condition, by the dominant hand, in both HSA and LSA (p < 0.05). The exemplar was manipulated significantly less in the dominant hand by the HSA group between the SH and H condition (p < 0.05). The exemplar was manipulated significantly less in the non‐dominant hand between the SH and H condition in both the HSA and LSA groups (p < 0.05). Error bars indicate ±1SD. H, haptic; HAT, haptic abilities test; HSA, high spatial abilities, LSA, low spatial abilities; n, frequency; p, p‐value; SD, standard deviation; SH, sighted haptic; %, percent. *p < 0.05.

FIGURE 7

Haptic strategies employed during SH and H conditions of the HAT in HSA and LSA groups (n = 22). The proportion of time (average percent) of response time each strategy was employed per question. The palpation strategy was used more in the LSA group compared to the HSA groups (p = 0.022). There were no other significant differences between groups. The palpation strategy was used significantly more in the SH condition in the HSA group (p = 0.013). The dynamic tracing and static hold strategy were used significantly less in the SH condition in both groups (p < 0.05). The lift strategy was used significantly less in the SH condition in the HSA group (p = 0.004). H, haptic; HAT, haptic abilities test; HSA, high spatial abilities, LSA, low spatial abilities; n, frequency; p, p‐value; SD, standard deviation; SH, sighted haptic; %, percent. *p < 0.05.

FIGURE 8

The Cognitive Theory of Multimedia Learning (CTML) words (red) and pictures (blue) are outlined in solid lines. In learning scenarios where touch is possible, the inclusion of haptic perception (green) provides viable sensory channel inputs for learning. The haptic input channel is appended to sensory input channels, when required, by the learner (outlined in broken line).