End-to-End Ultrasonic Hand Gesture Recognition

Abstract

As the number of electronic gadgets in our daily lives is increasing and most of them require some kind of human interaction, this demands innovative, convenient input methods. There are limitations to state-of-the-art (SotA) ultrasound-based hand gesture recognition (HGR) systems in terms of robustness and accuracy. This research presents a novel machine learning (ML)-based end-to-end solution for hand gesture recognition with low-cost micro-electromechanical (MEMS) system ultrasonic transducers. In contrast to prior methods, our ML model processes the raw echo samples directly instead of using pre-processed data. Consequently, the processing flow presented in this work leaves it to the ML model to extract the important information from the echo data. The success of this approach is demonstrated as follows. Four MEMS ultrasonic transducers are placed in three different geometrical arrangements. For each arrangement, different types of ML models are optimized and benchmarked on datasets acquired with the presented custom hardware (HW): convolutional neural networks (CNNs), gated recurrent units (GRUs), long short-term memory (LSTM), vision transformer (ViT), and cross-attention multi-scale vision transformer (CrossViT). The three last-mentioned ML models reached more than 88% accuracy. The most important innovation described in this research paper is that we were able to demonstrate that little pre-processing is necessary to obtain high accuracy in ultrasonic HGR for several arrangements of cost-effective and low-power MEMS ultrasonic transducer arrays. Even the computationally intensive Fourier transform can be omitted. The presented approach is further compared to HGR systems using other sensor types such as vision, WiFi, radar, and state-of-the-art ultrasound-based HGR systems. Direct processing of the sensor signals by a compact model makes ultrasonic hand gesture recognition a true low-cost and power-efficient input method.

Keywords: Fourier transform; HMI; MEMS ultrasonic transducer; machine learning; pre-processing.

Figure 1

Sensor-agnostic HGR processing flow.

Figure 2

Suggested change in HGR processing flow.

Figure 3

Data acquisition setup.

Figure 4

Processing shield with linear array; transducer locations are marked with yellow and white circles; yellow circle—sending transducer and white circle—receiving transducer.

Figure 5

Corner (left) and center (right) transducer array without processing shield; transducer locations marked with yellow and white circles; yellow circle—sending transducer and white circle—receiving transducer.

Figure 6

Gestures used in the dataset.

Figure 7

Part of a gesture frame of one channel of a pp gesture. Find an illustration of the pp gesture in Figure 6.

Figure 8

Plot of a pulse train with echo (in blue) compared to a pulse train without echo (in yellow) of transducer 1.

Figure 9

Plot of a pulse train with echo (in blue) compared to a pulse train without echo (in yellow) of transducer 2.

Figure 10

Plot of a pulse train with echo (in blue) compared to a pulse train without echo (in yellow) of transducer 3.

Figure 11

Schema of the CNN model with the best results. When the input is two channels (linear dataset), the output is 1 of 4 possible gestures. When the input is three channels (center and corner dataset), 1 out of 6 gestures is the output.

Figure 12

Accuracies per model per dataset.

Figure 13

Sizes of best models.

Figure 14

Best accuracy of the best models on all classes compared to the accuracy of the best models on four classes and the average accuracy of all models on each of the three datasets.

Figure 15

Accuracies per model averaged over datasets.