Interpreting Deep Learning Features for Myoelectric Control: A Comparison With Handcrafted Features

Abstract

Existing research on myoelectric control systems primarily focuses on extracting discriminative characteristics of the electromyographic (EMG) signal by designing handcrafted features. Recently, however, deep learning techniques have been applied to the challenging task of EMG-based gesture recognition. The adoption of these techniques slowly shifts the focus from feature engineering to feature learning. Nevertheless, the black-box nature of deep learning makes it hard to understand the type of information learned by the network and how it relates to handcrafted features. Additionally, due to the high variability in EMG recordings between participants, deep features tend to generalize poorly across subjects using standard training methods. Consequently, this work introduces a new multi-domain learning algorithm, named ADANN (Adaptive Domain Adversarial Neural Network), which significantly enhances (p = 0.00004) inter-subject classification accuracy by an average of 19.40% compared to standard training. Using ADANN-generated features, this work provides the first topological data analysis of EMG-based gesture recognition for the characterization of the information encoded within a deep network, using handcrafted features as landmarks. This analysis reveals that handcrafted features and the learned features (in the earlier layers) both try to discriminate between all gestures, but do not encode the same information to do so. In the later layers, the learned features are inclined to instead adopt a one-vs.-all strategy for a given class. Furthermore, by using convolutional network visualization techniques, it is revealed that learned features actually tend to ignore the most activated channel during contraction, which is in stark contrast with the prevalence of handcrafted features designed to capture amplitude information. Overall, this work paves the way for hybrid feature sets by providing a clear guideline of complementary information encoded within learned and handcrafted features.

Keywords: CNN; ConvNet; EMG; Grad-CAM; MAPPER; deep learning; feature extraction; gesture recognition.

Figure 1

Diagram of the workflow of this work. The 3DC Dataset is first preprocessed before being used to train the network using standard training and the proposed ADANN training procedure. The handcrafted features are directly calculated from the preprocessed dataset, while the deep features are extracted from the ConvNet trained with ADANN. In the diagram, the blue rectangles represent experiments and the arrows show which methods/algorithms are required to perform them.

Figure 2

The eleven hand/wrist gestures recorded in the 3DC Dataset (image re-used from Côté-Allard et al., 2019b).

Figure 3

The ConvNet's architecture, employing 543,629 learnable parameters. In this figure, Bi refers to the ith feature extraction block (i∈{1,2,3,4,5,6}). Conv refers to Convolutional layer. As shown, the feature extraction is performed after the non-linearity (leaky ReLU).

Figure 4

Overview of the training steps of ADANN (identical to DANN) for one labeled batch from the source ({x_s, y_s}, blue lines) and one unlabeled batch from the target ({x_t}, red dashed lines). The purple dotted lines correspond to the backpropagated gradient. The gradient reversal operation is represented by the purple diamond.

Figure 5

An example of step 3 of the Mapper algorithm with W = 2. The purple dots represent the elements of W. In (A), the red square corresponds to ℭ. In (B), ℭ is subdivided using k² squares of length H (with k = 2 in this case). The orange diamonds, in both (B,C), represent the elements of V. Finally, the square $c_{\overrightarrow{v}}$ of length D is shown on the upper left corner of (C), overlapping other squares centered on other elements of V (dotted lines).

Figure 6

Topological network generated exclusively for the handcrafted features, where nodes are colored to indicate percent composition of: (A) signal amplitude and power features (SAP), (B) non-linear complexity (NLC), (C) frequency information features (FI), (D) time series modeling features (TSM), and (E) unique features (UNI). Dashed boxes highlight dense groupings of the specified functional group in each of the networks.

Figure 7

Classification results of deep learning architectures. (A) Per-participant test set accuracy comparison when training the network with and without ADANN, (B) Confusion matrices on the test set for cross-subject training with and without ADANN.

Figure 8

Output of Guided Grad-CAM when asked to highlight specific gestures in an example. For all graphs, the y-axis of each channel are scaled to the same range of value (indicated on the first channel of each graph). Warmer colors indicate a higher “importance” of a feature in the input space for the requested gesture. The coloring use a logarithmic scale. For visualization purposes, only features that are within three order of magnitudes to the most contributing feature are colored. (A) The examples shown are real examples and correspond to the same gestures that Guided Grad-CAM is asked to highlight. (B) A single example, generated using Gaussian noise of mean 0 and standard deviation 450, is shown three times. While the visualization algorithm does highlight features in the input space (when the requested gesture is not truly present in the input), the magnitude of these contributions is substantially smaller (half or less) than when the requested gesture is present in the input.

Figure 9

Topological network generated for exclusively the learned features, where nodes are colored to indicate percent composition of: (A) Block 1's features, (B) Block 2's features, (C) Block 3's features, (D) Block 4's features, (E) Block 5's features, and (F) Block 6's features. Dashed boxes highlight dense groupings of the specified block features in each of the networks.

Figure 10

Topological network generated for all features, where nodes were colored to indicate percent composition of learned features. The dashed boxes highlight dense grouping of handcrafted features with their associated type.

Figure 11

Confusion matrices using the handcrafted features and the learned features from the first, penultimate and last block as input and a LDA as the classifier. The first column, denoted as All features, shows the confusion matrices when using all 64 learned features of Block 1, 5, and 6, respectively (from top to bottom) and the set of UNI handcrafted features. The next five columns, denoted as Single Feature, show the confusions matrices for handcrafted feature examplars and from the same network's blocks but when training the LDA on a single feature. The subset of learned features was selected as representative of the typical confusion matrices found at each block. The examplars of the handcrafted features were selected from each handcrafted features' category (in order: SAP, FI, NLC, TSM, and UNI).

Figure 12

Mean squared error of the regressions from learned features to handcrafted features, with respect to the number of blocks employed for the regression. The features are grouped with their respective functional groups.