Counting Finger and Wrist Movements Using Only a Wrist-Worn, Inertial Measurement Unit: Toward Practical Wearable Sensing for Hand-Related Healthcare Applications

Abstract

The ability to count finger and wrist movements throughout the day with a nonobtrusive, wearable sensor could be useful for hand-related healthcare applications, including rehabilitation after a stroke, carpal tunnel syndrome, or hand surgery. Previous approaches have required the user to wear a ring with an embedded magnet or inertial measurement unit (IMU). Here, we demonstrate that it is possible to identify the occurrence of finger and wrist flexion/extension movements based on vibrations detected by a wrist-worn IMU. We developed an approach we call "Hand Activity Recognition through using a Convolutional neural network with Spectrograms" (HARCS) that trains a CNN based on the velocity/acceleration spectrograms that finger/wrist movements create. We validated HARCS with the wrist-worn IMU recordings obtained from twenty stroke survivors during their daily life, where the occurrence of finger/wrist movements was labeled using a previously validated algorithm called HAND using magnetic sensing. The daily number of finger/wrist movements identified by HARCS had a strong positive correlation to the daily number identified by HAND (R² = 0.76, p < 0.001). HARCS was also 75% accurate when we labeled the finger/wrist movements performed by unimpaired participants using optical motion capture. Overall, the ringless sensing of finger/wrist movement occurrence is feasible, although real-world applications may require further accuracy improvements.

Keywords: convolutional neural network (CNN); human activity recognition (HAR); motion capture system (MC); neural network; rehabilitation; stroke; the inertial measurement unit (IMU); wearable sensing.

Figure 1

(A) Marker placements for inferring the wrist and finger angle. $\alpha$ represents the wrist angle. Additionally, $\beta$ represents the finger angle (i.e., metacarpophalangeal joint angle). Four markers were taped as shown in the figure to the wrist and finger to obtain $\alpha$ and $\beta$. (B) The list of movements in the Mocap-Lab Dataset. Subjects performed 6 movements involving an arm-only movement, hand-only movement, and hand and arm movement.

Figure 2

The 6 steps for training and assessing the neural network.

Figure 3

An example of processing a data-sample where the HAND algorithm predicted the existence of a hand movement. This data-sample contained 150 time-samples from the IMU. (A) Acceleration, gyro, and gravity vector measures are shown in each row as a function of time, with the columns indicating the x, y, and z axes of the sensor. (B) The heatmaps of the spectrograms computed from this sample for each measurement. (C) The heatmaps of the normalized spectrograms.

Figure 4

An example of the CNN architecture trained by the Manumeter-Home Dataset.

Figure 5

(A) Accuracies and AUCs for HARCS-based identification of finger/wrist movements in-the-wild for 20 stroke subjects with varying UEFM scores. (B) The correlation between HARCS and HAND counts of finger/wrist movement occurrence, where HAND counts were produced in a previous study using information from a magnetic ring.

Figure 6

The relationship between the network performance and the mean linear acceleration across the data-sample window of duration ~2.8 s. (A) Number of data samples segmented by mean acceleration. (B–E) Performance metric segmented by the mean acceleration. (B) Accuracy. (C) Precision, (D) Recall. (E) F1-score. (F,G) Proportion of classifications segmented by the mean acceleration. (F) False positive rate. (G). False negative rate. In (A), the proportions of actual positives and total data-samples are shown on the top of each bar.

Figure 7

Comparison of HARC algorithm counts versus HAND counts and the true number of hand movements for (A) Hand-only exercises by people with a stroke (B) Hand-only exercises by unimpaired subjects, (C) Arm-only exercise by unimpaired subjects. For (A,B), perfect counting resulted in 100% movement counts. For (C), perfect counting resulted in 0% movement counts, since C was arm-only exercise. A value of 100% represents 50 hand movements for the hand-only exercise and 200 movements for the arm-only exercise.

Figure 8

(A,B) Average accuracy of HARCS when trained and tested on the Mocap-Lab Dataset obtained from nine unimpaired subjects. (C,D) The mean of the ROC Curve (solid line) in which the shaded area represents $\pm$ 1 standard deviation. In (A,C), combined hand/arm movements were treated as an actual positive. In (B,D), combined hand/arm movements were treated as an actual negative.