Digit-tracking as a new tactile interface for visual perception analysis

Abstract

Eye-tracking is a valuable tool in cognitive science for measuring how visual processing resources are allocated during scene exploration. However, eye-tracking technology is largely confined to laboratory-based settings, making it difficult to apply to large-scale studies. Here, we introduce a biologically-inspired solution that involves presenting, on a touch-sensitive interface, a Gaussian-blurred image that is locally unblurred by sliding a finger over the display. Thus, the user's finger movements provide a proxy for their eye movements and attention. We validated the method by showing strong correlations between attention maps obtained using finger-tracking vs. conventional optical eye-tracking. Using neural networks trained to predict empirically-derived attention maps, we established that identical high-level features hierarchically drive explorations with either method. Finally, the diagnostic value of digit-tracking was tested in autistic and brain-damaged patients. Rapid yet robust measures afforded by this method open the way to large scale applications in research and clinical settings.

Fig. 1

Digit-tracking method for recording ocular exploration via a connected touch-sensitive interface. a The resolution of the displayed picture is reduced using Gaussian blur, simulating peripheral vision, while in an area above the tactile contact point the central vision is simulated by revealing the picture in native resolution through a Gaussian aperture window. Subjects can explore the picture by moving their digit over the display. b Example of parameterization of the apparatus for real-time recording (orange curve). Two parameters are set: the size of the aperture window (α) and the maximum level of detail perceptible in the simulated peripheral vision (β). Optimal exploration is obtained using a simulated peripheral acuity close to the real human peripheral acuity (black curve–from ref. ) and a central vision slightly narrower than the foveal region (see Methods section for a complete description of the strategy for optimal adjustment of these parameters). c Average heat maps representing the picture’s normalized probability density of exploration (scaled between 0 and 1), estimated by two independent groups of subjects (each group N = 11) using either eye-tracking (left) or digit-tracking (right) methods. The two estimates are highly correlated (r_pearson = 0.73) and show similar high attentional saliency regions.

Fig. 2

Digit and eye-tracking comparison. A total of 122 pictures featuring humans, animals, objects or abstract art were divided in two sets (Set A and Set B). Half of the subjects explored Set A using digit-tracking and Set B with eye-tracking, while the other half explored Set B using digit-tracking and Set A using eye-tracking, thus yielding two independent sets of attention maps for each method. a Violin plots of the Pearson’s correlation coefficients calculated, for each picture, between the probability density estimates of exploration measured with eye-tracking or digit-tracking (11 subjects/condition). Both techniques can measure precise attention maps that are highly correlated (Set A: median correlation = 0.71, sign-test p < 9 × 10⁻¹⁹; Set B: median correlation = 0.72, p < 9 × 10⁻¹⁹; no significant differences between image sets: Wilcoxon p = 0.72). b, c Inter-subject correlations scatter and violin plots, respectively. Inter-subject correlations were calculated, for each image/technology couple, as the average Pearson correlation coefficient between the exploration densities of one subject with the exploration density of another subject. ISC calculated with eye-tracking or digit-tracking are correlated (B) (r = 0.42, p < 2 × 10⁻⁶). No significant differences can be found for ISC between explorations measured with eye-tracking or digit-tracking (c) (median_eyes = 0.44, median_digit = 0.43, Wilcoxon p = 0.32). d Convergence of exploration normalized density estimates. Each curve represents for one image/technology couple recorded with N subjects, the percent of variance of the exploration normalized density estimates that could be explained with N-1 subject. e Violin plots of the number of subjects necessary to explain more than 95% of variance. Eye-tracking and digit-tracking show similar performances (median = 5, seven subjects are sufficient to obtain stable measurements on most images).

Fig. 3

Convolutional Neural network computation of saliency maps. A model of artificial vision was used to predict the most salient areas in an image and test whether attention maps derived from digit-tracking and eye-tracking exploration data are sensitive to the same features in visual scenes. a Convolutional Neural Network architecture. The first five convolutional layers of the AlexNet Network were used for features map extraction, and features are linearly combined in the last layer to produce saliency maps (e.g., Fig. S4). b Hierarchical ordering of learned weights in the last layer of the convolutional neural network (CNN). X-axis denotes the 256 outputs while the Y-axis denotes the mean Pearson correlation between an individual channel and the measured saliency map. Each channel can be seen as a saliency map sensitive to a single feature class in the picture. A strong positive correlation coefficient indicates a highly attractive feature while a strong negative correlation indicates a highly avoided feature. c Correlation between the weights learned using eye and digit-tracking (Set A r_pearson = 0.95, p < 1 × 10⁻¹²⁸–Set B r_pearson = 0.96, p < 1 × 10⁻¹⁴⁷). d High-level features are visualized by identifying in the picture database the most responsive pixels for the considered CNN channel. Example of the most attractive and the most avoided features corresponding to, respectively, the 3 most positively correlated and the 3 most negatively correlated channels of the CNN. Human explorations are particularly sensitive to eyes or eye-like areas, faces and highly contrasted details, while uniform areas with natural colors, textures, repetitive symbols are generally avoided.

Fig. 4

Atypical exploration behaviors recorded using digit-tracking. a Example of image exploration by a patient suffering from hemispatial neglect (Hemi. Neglect) due to right-parietal lobe damage following stroke and a control subject. The right panels represent the average spatial distribution of the explorations recorded for 32 images in a single examination session. The spatial attention bias can be precisely quantified: for the control subject, 49.6% of their exploration is on the right side of the display (corresponding to a Z-score of −0.1, centile 45, according to a reference population, N = 22), whereas for the patient, 82.7% of the exploration has been recorded on the right side of the display (corresponding to a Z-score of 9.8, centile > 99.999). b Exploration of an image with social content by a 14-year-old non-verbal autistic child compared to a neurotypical control subject. The patient adopts an exploration strategy which avoids human faces (red frame), whereas these are the most explored scene elements in the control population. Please note that original faces have been modified to hide individuals’ identity.

Fig. 5

Detection of abnormal visual explorations in autistic spectrum disorders (ASD). a Typical exploration behaviors observed in four subjects (two ASD and two control subjects), recorded with either eye-tracking or digit-tracking. High-functioning autistic patients (ASD) tend to adopt an atypical face exploration strategy that avoids the eye area. Note similarities in the explorations, even if they are recorded with different methods on different subjects. b Results of the group analyses: For each subject, a single score was calculated to quantify the neurotypicity of attention maps obtained in patients (N = 22) and control subjects (N = 22). Both methods (Eye-tracking and digit-tracking) detect anomalies in the attention maps of the ASD population (respectively p_eye < 0.0001 and p_digit < 0.00001–Wilcoxon rank-sum test) and are correlated (Spearman rho = 0.56 p < 0.0002). c Receiver Operating Characteristic (ROC) curves for ASD/CTRL classifications with the ‘exploration-neurotypicality score’. Please note that original faces have been modified to hide individuals’ identity.