Real-Time Human Activity Recognition with IMU and Encoder Sensors in Wearable Exoskeleton Robot via Deep Learning Networks

Abstract

Wearable exoskeleton robots have become a promising technology for supporting human motions in multiple tasks. Activity recognition in real-time provides useful information to enhance the robot's control assistance for daily tasks. This work implements a real-time activity recognition system based on the activity signals of an inertial measurement unit (IMU) and a pair of rotary encoders integrated into the exoskeleton robot. Five deep learning models have been trained and evaluated for activity recognition. As a result, a subset of optimized deep learning models was transferred to an edge device for real-time evaluation in a continuous action environment using eight common human tasks: stand, bend, crouch, walk, sit-down, sit-up, and ascend and descend stairs. These eight robot wearer's activities are recognized with an average accuracy of 97.35% in real-time tests, with an inference time under 10 ms and an overall latency of 0.506 s per recognition using the selected edge device.

Keywords: deep learning networks; encoders; inertial measurement unit; real-time human activity recognition; wearable exoskeleton robot.

Figure 1

The wearable robot and implemented HAR system: (a) Wearable exoskeleton robot, (b) Data collection and preprocesses, (c) Deep learning models for HAR, (d) Hardware platforms, and (e) Recognized activities.

Figure 2

A scenario for continuous human activities of the robot wearer.

Figure 3

Deep Learning model structure: (a) CNN, (b) RNN, (c) LSTM, (d) BiLSTM, (e) GRU.

Figure 4

A sample confusion matrix with the Bi-LSTM-2L model and epoch dataset.

Figure 5

Continuous real-time HAR results from two subjects: (a) S₁ and (b) S₂.

Figure 6

Recognition examples and screenshots with HAR system results during continuous real-time tests.