Hand Gesture Recognition Using Single Patchable Six-Axis Inertial Measurement Unit via Recurrent Neural Networks

Abstract

Recording human gestures from a wearable sensor produces valuable information to implement control gestures or in healthcare services. The wearable sensor is required to be small and easily worn. Advances in miniaturized sensor and materials research produces patchable inertial measurement units (IMUs). In this paper, a hand gesture recognition system using a single patchable six-axis IMU attached at the wrist via recurrent neural networks (RNN) is presented. The IMU comprises IC-based electronic components on a stretchable, adhesive substrate with serpentine-structured interconnections. The proposed patchable IMU with soft form-factors can be worn in close contact with the human body, comfortably adapting to skin deformations. Thus, signal distortion (i.e., motion artifacts) produced for vibration during the motion is minimized. Also, our patchable IMU has a wireless communication (i.e., Bluetooth) module to continuously send the sensed signals to any processing device. Our hand gesture recognition system was evaluated, attaching the proposed patchable six-axis IMU on the right wrist of five people to recognize three hand gestures using two models based on recurrent neural nets. The RNN-based models are trained and validated using a public database. The preliminary results show that our proposed patchable IMU have potential to continuously monitor people's motions in remote settings for applications in mobile health, human-computer interaction, and control gestures recognition.

Keywords: control gestures; hand gesture recognition; patchable IMU; recurrent neural network; six-axis inertial sensor.

Figure 1

Design of patchable inertial measurement unit (IMU). (a) Reconfigurable system concept and schematic; (b) system block diagram.

Figure 2

Implementation process patchable IMU. (a) Design and formulation of layout footprint; (b) water-soluble tape (WST)/polyimide film attachment and thermal release tape (TRT) removal; (c) Soldering and top-layer encapsulation; (d) Detachment of WST/polyimide film; (e) Bottom-layer encapsulation; (f) Battery and module alignment.

Figure 3

Recurrent units. (a) Long short-term memory; (b) gate recurrent unit.

Figure 4

Hand gestures classifier based on a recurrent neural network (RNN) with bidirectional long short term memory (BiLSTM).

Figure 5

Hand gestures classifier based on RNN with GRU.

Figure 6

Implementation results. (a) Wireless epidermal six-axis IMU; (b) waterproof experimental results; (c) stretchability test results with 30% tensile strain; (d) wrist bending experimental results after first, 50th and 100th repetitions; (e) Bluetooth low energy (BLE) paring antenna measurement result in free space.

Figure 6

Implementation results. (a) Wireless epidermal six-axis IMU; (b) waterproof experimental results; (c) stretchability test results with 30% tensile strain; (d) wrist bending experimental results after first, 50th and 100th repetitions; (e) Bluetooth low energy (BLE) paring antenna measurement result in free space.

Figure 7

Precision and recall with the public database. (a) RNN-BiLSTM; (b) RNN-gate recurrent unit (GRU).

Figure 8

Precision and recall with the collected data. (a) RNN-BiLSTM; (b) RNN-GRU.