Haptic exploratory behavior during object discrimination: a novel automatic annotation method

Abstract

In order to acquire information concerning the geometry and material of handheld objects, people tend to execute stereotypical hand movement patterns called haptic Exploratory Procedures (EPs). Manual annotation of haptic exploration trials with these EPs is a laborious task that is affected by subjectivity, attentional lapses, and viewing angle limitations. In this paper we propose an automatic EP annotation method based on position and orientation data from motion tracking sensors placed on both hands and inside a stimulus. A set of kinematic variables is computed from these data and compared to sets of predefined criteria for each of four EPs. Whenever all criteria for a specific EP are met, it is assumed that that particular hand movement pattern was performed. This method is applied to data from an experiment where blindfolded participants haptically discriminated between objects differing in hardness, roughness, volume, and weight. In order to validate the method, its output is compared to manual annotation based on video recordings of the same trials. Although mean pairwise agreement is less between human-automatic pairs than between human-human pairs (55.7% vs 74.5%), the proposed method performs much better than random annotation (2.4%). Furthermore, each EP is linked to a specific object property for which it is optimal (e.g., Lateral Motion for roughness). We found that the percentage of trials where the expected EP was found does not differ between manual and automatic annotation. For now, this method cannot yet completely replace a manual annotation procedure. However, it could be used as a starting point that can be supplemented by manual annotation.

Fig 1. Stimuli used during haptic discrimination.

Because of different mid and outer layers, all differed in roughness, hardness, volume, and weight. Stimulus B contains an actual sensor, while stimuli A and C contain dummies.

Fig 2. Sensor placement.

In total, seven 6 DoF electromagnetic sensors were used during the data gathering procedure. Three on each hand and one inside the stimulus.

Fig 3. Manual annotation setup.

The top monitor shows the recorded video in fullscreen while the lower monitor enables annotation and playback options.

Fig 4. Annotation of two example exploration trials.

Panel A: the left and right columns show output from all raters for a volume and roughness trial respectively. The Possible EPs are: Pressure (PR), Lateral Motion (LM), Unsupported Holding (UH), and Enclosure (EN). The width of the box represents the normalized trial duration and each annotated interval is color coded according to the legend. Panel B shows the main and secondary EP based on total duration annotated by each rater. The same legend is used to indicate the EP and hand. Panel C displays the pairwise agreement as a percentage of trial duration.

Fig 5. Mean percentage of trial duration that a pair of raters is in agreement over the annotation of all EPs.

Data are shown for each property separately and as the mean over all trials. The green dashed line indicates mean agreement between human and random annotation. Error bars represent standard deviation.

Fig 6. Percentage of trials for which the expected EP was found as the main or secondary EP.

The expected links are: Lateral Motion for roughness, Pressure for hardness, Unsupported Holding for weight, and Enclosure for volume. This percentage is given as a function of object property and rater type. Error bars represent standard deviation.