A Novel Framework Based on Deep Learning Architecture for Continuous Human Activity Recognition with Inertial Sensors

Abstract

Frameworks for human activity recognition (HAR) can be applied in the clinical environment for monitoring patients' motor and functional abilities either remotely or within a rehabilitation program. Deep Learning (DL) models can be exploited to perform HAR by means of raw data, thus avoiding time-demanding feature engineering operations. Most works targeting HAR with DL-based architectures have tested the workflow performance on data related to a separate execution of the tasks. Hence, a paucity in the literature has been found with regard to frameworks aimed at recognizing continuously executed motor actions. In this article, the authors present the design, development, and testing of a DL-based workflow targeting continuous human activity recognition (CHAR). The model was trained on the data recorded from ten healthy subjects and tested on eight different subjects. Despite the limited sample size, the authors claim the capability of the proposed framework to accurately classify motor actions within a feasible time, thus making it potentially useful in a clinical scenario.

Keywords: activities of daily living; artificial intelligence; bioengineering; convolutional neural networks; data augmentation; deep learning; human activity recognition; inertial measurement units; motion analysis; rehabilitation; time-series.

Figure 1

The framework oriented to CHAR: inertial data are collected from four sensors placed on the human body; each component of the IMU sensors is used to feed a multi-branch DL-based architecture; this model is trained on the data related to a separate execution of four ADLs that are Lying-down (LD), Sit-to-stand (S2S), Walking (W), and Turning (TN); then, it is tested on the signals coming from motor actions that are performed uninterruptedly.

Figure 2

The placement chosen for the sensor in the proposed framework: two IMUs are located on the two sides of the human pelvis to monitor motor actions driving the lower limbs (e.g., walking, sit-to-stand, lying-down); one sensor on the sternum serves to register the trunk contribution to accomplishing sit-to-stand and lying-down activities; the sensor on the right wrist (i.e., the wrist of the dominant arm) aims at acquiring the possible use of hands during lying-down, as well as the arm swing while walking.

Figure 3

Motion Studio system with IMU sensors, Access Point, Docking Station, and PC.

Figure 4

The two experimental protocols encompassed in the study with the aim of continuous HAR: (a) the protocol for acquiring the training dataset encompasses an interrupted execution of motor tasks, which are Lying-down, Sit-to-stand, and Walking+Turning; (b) the protocol for collecting the test dataset requires subject to perform the same tasks without interruptions on a predefined path.

Figure 5

Data augmentation technique to cope with operational differences.

Figure 6

The custom multi-branch CNN addressing CHAR.

Figure 7

Boxplots of accuracy distributions on the test set for each sensor combination, with *, **, and *** representing statistically significant comparisons with $p<0.05$, $p<0.01$, and $p<0.001$, respectively.

Figure 8

Confusion matrices of the proposed model for continuous human activity recognition for three sensor configurations: (a) RW+LP+S means that inertial sensors are placed at the right wrist, the left pelvis, and the sternum; (b) LP+S means that inertial sensors are placed at the left pelvis and the sternum; (c) RW+S means that inertial sensors are placed at the right wrist and the sternum; (d) RW+RP+LP means that inertial sensors are placed at the right wrist, and the right and left pelvises; (e) RP+LP means that inertial sensors are placed at the right pelvis and left pelvis; (f) RW+RP means that inertial sensors are placed at the right wrist and the right pelvis; (g) RW+RP+LP+S means that inertial sensors are placed at the right wrist, the right and left pelvises, and the sternum; (h) RW+RP+S means that inertial sensors are placed at the right wrist, the right pelvis, and the sternum; (i) RP+LP+S means that inertial sensors are placed at the right and left pelvises and the sternum. The activities to be recognized are Lying-down (LD), Sit-to-stand (S2S), Turning (TN), and Walking (W).

Figure 9

Boxplots of inference time distributions on the test set for each sensor combination, with * and ** representing statistically significant comparisons with $p<0.05$ and $p<0.01$, respectively.