Uneven Terrain Recognition Using Neuromorphic Haptic Feedback

Abstract

Recent years have witnessed relevant advancements in the quality of life of persons with lower limb amputations thanks to the technological developments in prosthetics. However, prostheses that provide information about the foot-ground interaction, and in particular about terrain irregularities, are still missing on the market. The lack of tactile feedback from the foot sole might lead subjects to step on uneven terrains, causing an increase in the risk of falling. To address this issue, a biomimetic vibrotactile feedback system that conveys information about gait and terrain features sensed by a dedicated insole has been assessed with intact subjects. After having shortly experienced both even and uneven terrains, the recruited subjects discriminated them with an accuracy of 87.5%, solely relying on the replay of the vibrotactile feedback. With the objective of exploring the human decoding mechanism of the feedback startegy, a KNN classifier was trained to recognize the uneven terrains. The outcome suggested that the subjects achieved such performance with a temporal dynamics of 45 ms. This work is a leap forward to assist lower-limb amputees to appreciate the floor conditions while walking, adapt their gait and promote a more confident use of their artificial limb.

Keywords: FPGA neuron model; Izhikevich; PSTH-based classification; lower-limb impairments; neuromorphic haptic feedback; tactile augmentation; terrain recognition; wearable assistive robotics.

Figure 1

Experimental setup overview. (a) Familiarization phase: Exploration of the three terrains, i.e., tiles, grass and stones (terrain configurations in the pictures). The detection of gait events by the insole activates the corresponding VTs as follows: VT1 is triggered at the heel–strike (HS); VT2 is triggered at the foot-flat (FF); and VT3 is triggered at the toe-off (TO). (b) The wearable augmenting haptic belt embedding the VTs placed along the waist from the spine (VT1) to the navel (VT3). (c) Neuromorphic vibrotactile feedback: examples of real-time spike trains delivered by each VT unit according to the detected stance phase while walking along even (grass and tiles) and uneven (stones) floors. (d) Neuromorphic vibrotactile feedback computation: example of the activation of VT1 relying on the foot pressure sensors embedded in the insole and the relevant neuromorphic computation.

Figure 2

The customized fixed-point pipelined architecture designed for the Izhikevich neuron. The InVTi, BRAM_V and BRAM_U store the values of the input, $I_{VTi}^n$, v and u, respectively. The red numbers represent the fixed-point representation at every computational unit and the dotted lines denote the computational cycles.

Figure 3

Workflow of the algorithm decoding. First box: the spike trains of each trial (i.e., 5 stances over a terrain) are pre-processed into the PSTHs for every bin size, ranging from 0% to 50% of the stance cycle. Two examples, for 1.7% (left column) and 5.1% (right column) bin sizes, are reported. The last activation time of VT1 (Feature 1) and the activation times of VT2 and VT3 (Feature 2 and Feature 3, respectively) are extracted and used as input features for the KNN algorithm. The second box represents the KNN input feature space for the two bin sizes. The last box shows the confusion matrix of the terrain classification task that the KNN algorithm outputs at each bin size.

Figure 4

Terrain recognition and identification during playback: (a) confusion matrix of the subjects’ uneven terrain recognition; (b) confusion matrix of the subjects’ terrain identification; (c,d) accuracy_H and Clopper–Pearson exact intervals (error bars) for even/uneven terrain recognition and for each terrain type identification, respectively.

Figure 5

Population-wise algorithm decoding performance. Unevenness recognition performance (left) and three-terrain identification performance (right). Top: accuracy_A as a function of the bin interval measured as percentage of the stance duration (solid line); it is compared with subjects’ accuracy (shaded CI), with the chance level (flat solid line) and with the candidacy (right y-axis, dotted line). Bottom: confusion matrices of the algorithm classification output at the bin size corresponding to the maximum candidacy.

Figure 6

Subject-wise decoding performance. Unevenness recognition (left) and three-terrain identification performance (right). Top: accuracy_A (solid line) as a function of the bin size measured as percentage of the stance duration; it is compared with the subjects’ accuracy (shaded CI), with the chance level (flat solid line, 50% for the unevenness recognition and 33% for the terrain identification) and with the candidacy (right y-axis, dotted line). Bottom: averaged confusion matrices and accuracy_A of the algorithm classification output at the bin size corresponding to the maximum candidacy for each individual subject.

Figure 7

Effect of VT2 on algorithm performance. The maximum candidacy and its CI for VT combinations with (+) and without (−) VT2 for each subject and all the subjects grouped together are represented. The maximum candidacy when all the input VTs are considered is shown in pink, as reference. The presence of VT2 returned similar results to the all VTs cases (pink data) for unevenness recognition ((a), purple data) and three-terrain identification ((b), blue data) in most of the cases.