Hand Gesture Recognition on Edge Devices: Sensor Technologies, Algorithms, and Processing Hardware

Abstract

Hand gesture recognition (HGR) is a convenient and natural form of human-computer interaction. It is suitable for various applications. Much research has already focused on wearable device-based HGR. By contrast, this paper gives an overview focused on device-free HGR. That means we evaluate HGR systems that do not require the user to wear something like a data glove or hold a device. HGR systems are explored regarding technology, hardware, and algorithms. The interconnectedness of timing and power requirements with hardware, pre-processing algorithm, classification, and technology and how they permit more or less granularity, accuracy, and number of gestures is clearly demonstrated. Sensor modalities evaluated are WIFI, vision, radar, mobile networks, and ultrasound. The pre-processing technologies stereo vision, multiple-input multiple-output (MIMO), spectrogram, phased array, range-doppler-map, range-angle-map, doppler-angle-map, and multilateration are explored. Classification approaches with and without ML are studied. Among those with ML, assessed algorithms range from simple tree structures to transformers. All applications are evaluated taking into account their level of integration. This encompasses determining whether the application presented is suitable for edge integration, their real-time capability, whether continuous learning is implemented, which robustness was achieved, whether ML is applied, and the accuracy level. Our survey aims to provide a thorough understanding of the current state of the art in device-free HGR on edge devices and in general. Finally, on the basis of present-day challenges and opportunities in this field, we outline which further research we suggest for HGR improvement. Our goal is to promote the development of efficient and accurate gesture recognition systems.

Keywords: 4G; 5G; AI accelerators; LTE; WiFi; algorithms; artificial intelligence; edge machine learning; hand gesture recognition; image processing; lidar; radar; signal processing; ultrasound; vision.

Figure 1

Applications of gesture recognition and similar systems.

Figure 2

Technology-agnostic processing flow.

Figure 3

Synergy of requirements, technology, algorithms for HGR systems.

Figure 4

Actuation with a single frequency.

Figure 5

Actuation with a chirp.

Figure 6

Actuation for UWB.

Figure 7

Spectrogram (in radar often called micro-doppler), calculated by 1D Fourier transform in fast-time direction, over the entire sample in a frequency band around the sent single frequency.

Figure 8

4 ultrasound sender at ${\displaystyle \frac{\lambda }{2}}$ spacing, 24.5 kHz frequency, 0° steering.

Figure 9

4 ultrasound sender at ${\displaystyle \frac{\lambda }{2}}$ spacing, 24.5 kHz frequency, 45° steering.

Figure 10

8 ultrasound sender at ${\displaystyle \frac{\lambda }{2}}$ spacing, 24.5 kHz frequency, 0° steering.

Figure 11

8 ultrasound sender at ${\displaystyle \frac{\lambda }{2}}$ spacing, 24.5 kHz frequency, 45° steering.

Figure 12

Raw ultrasound data sample in 3D.

Figure 13

Raw ultrasound data sample in 2D only segmented part plotted.

Figure 14

Raw ultrasound data sample in 2D.

Figure 15

Range-doppler/velocity map, calculated by 2D Fourier transform (in fast - first and then in slow-time direction), of the segmented gesture.

Figure 16

HGR classification algorithms and processes.