Multi-modality deep forest for hand motion recognition via fusing sEMG and acceleration signals

Abstract

Bio-signal based hand motion recognition plays a critical role in the tasks of human-machine interaction, such as the natural control of multifunctional prostheses. Although a large number of classification technologies have been taken to improve the motion recognition accuracy, it is still a challenge to achieve acceptable performance for multiple modality input. This study proposes a multi-modality deep forest (MMDF) framework to identify hand motions, in which surface electromyographic signals (sEMG) and acceleration signals (ACC) are fused at the input level. The proposed MMDF framework constitutes of three main stages, sEMG and ACC feature extraction, feature dimension reduction, and a cascade structure deep forest for classification. A public database "Ninapro DB7" is used to evaluate the performance of the proposed framework, and the experimental results show that it can achieve a significantly higher accuracy than that of competitors. Besides, our experimental results also show that MMDF outperforms other traditional classifiers with the input of the single modality of sEMG signals. In sum, this study verifies that ACC signals can be an excellent supplementary for sEMG, and MMDF is a plausible solution to fuse mulit-modality bio-signals for human motion recognition.

Keywords: Acceleration signals; Deep forest; Hand gesture classification; sEMG signals.

Fig. 1

The diagram of hand motion recognition framework, where signals reprocessing, feature extraction, feature dimension reduction, and MMDF classifiers are included

Fig. 2

The structure of feature extraction; The feature extraction process of the sEMG and the ACC has a lot in common, and it is necessary to perform window segmentation on the data of each channel separately, and then extract the features. The difference is that five features of MAV, WL, AR4, SSC, and ZC are extracted for the sEMG, and only the average value is extracted for the ACC. Finally, the two data types are merged into one feature vector

Fig. 3

The structure of cascade forest; each layer of the cascade consists of two types of random forests with different color blocks, and the number of layers is optimised during the training stage

Fig. 4

Sensor placement on able-bodied (left) and amputee subjects (centre, right) [20]. Eight EMG-IM sensors are equally spaced around the participants’forearm (3 cm below the elbow). Two are place on EDC and FDS muscles, and the rest two on biceps and triceps muscles. Elastic bandage is used to fix sensors

Fig. 5

The change of classification accuracy when the use of different number of trees in each forest

Fig. 6

The above figure respectively shows the classification accuracy of the intact subject and the amputee subject for three different data input EMG+ACC, EMG, ACC, where A is the experimental result before dimensionality reduction, B is the result of dimensionality reduction processing on the ACC features using PCA. For intact subjects, each bar shows the average classification accuracy across 20 complete subjects for 40 types of gestures. For amputation subjects, each bar shows the average accuracy of two amputees, where 38 motions are classified

Fig. 7

The classification accuracy comparison among MMDF (this study), RF, SVM, KNN and the original GcForest across all different subjects, including two amputees s21 and s22

Fig. 8

The Confusion matrices obtained from subject s1 (intact) and subject s21 (amputee), and are displayed in the left panel and the right panel, respectively. The x-axis of the matrices is the class label of the predicted results by the proposed methods MMDF, while the y-axis shows the actual class label. The change of color shows the accuracy

Fig. 9

The comparison of time cost on different classifiers