Deep Learning-Based Upper Limb Functional Assessment Using a Single Kinect v2 Sensor

Abstract

We develop a deep learning refined kinematic model for accurately assessing upper limb joint angles using a single Kinect v2 sensor. We train a long short-term memory recurrent neural network using a supervised machine learning architecture to compensate for the systematic error of the Kinect kinematic model, taking a marker-based three-dimensional motion capture system (3DMC) as the golden standard. A series of upper limb functional task experiments were conducted, namely hand to the contralateral shoulder, hand to mouth or drinking, combing hair, and hand to back pocket. Our deep learning-based model significantly improves the performance of a single Kinect v2 sensor for all investigated upper limb joint angles across all functional tasks. Using a single Kinect v2 sensor, our deep learning-based model could measure shoulder and elbow flexion/extension waveforms with mean CMCs >0.93 for all tasks, shoulder adduction/abduction, and internal/external rotation waveforms with mean CMCs >0.8 for most of the tasks. The mean deviations of angles at the point of target achieved and range of motion are under 5° for all investigated joint angles during all functional tasks. Compared with the 3DMC, our presented system is easier to operate and needs less laboratory space.

Keywords: Kinect; deep learning; kinematics; recurrent neural network; upper limb functional assessment.

Figure 1

The architecture of our deep learning refined kinematic model for Kinect v2.

Figure 2

The kinematic models of the Kinect v2 system and the 3D motion capture system.

Figure 3

Illustration of the Kinect skeleton joints and the anatomical coordination system.

Figure 4

Marker set for the 3DMC system. Left: Arrangement of the UWA upper limb marker set. Right: A participant with the attached markers.

Figure 5

Architecture of our LSTM neural network for upper limb kinematics refinement.

Figure 6

Four upper limb functional tasks evaluated in our study. Left: Hand to the contralateral shoulder. Middle-left: Hand to mouth or drinking. Middle-right: combing hair. Right: Hand to back pocket.

Figure 7

The protocol of the leave one subject out cross validation (LOOCV).

Figure 8

Joint angles during the hand to contra lateral shoulder task calculated via the kinematic model for 3DMC (orange solid line), our deep learning refined kinematic model for Kinect (green dashed line) and the kinematic model for Kinect (blue solid line). The joint angles include shoulder flexion (+)/extension (−), shoulder adduction (+)/abduction (−), shoulder internal rotation (+)/external rotation (−), and elbow flexion (+)/extension (−).

Figure 9

Joint angles during the hand to mouth task calculated via the kinematic model for 3DMC (orange solid line), our deep learning refined kinematic model for Kinect (green dashed line) and the kinematic model for Kinect (blue solid line). The joint angles include shoulder flexion (+)/extension (−), shoulder adduction (+)/abduction (−), shoulder internal rotation (+)/external rotation (−), and elbow flexion (+)/extension (−).

Figure 10

Joint angles during the combing hair task calculated via the kinematic model for 3DMC (orange solid line), our deep learning refined kinematic model for Kinect (green dashed line) and the kinematic model for Kinect (blue solid line). The joint angles include shoulder flexion (+)/extension (−), shoulder adduction (+)/abduction (−), shoulder internal rotation (+)/external rotation (−), and elbow flexion (+)/extension (−).

Figure 11

Joint angles during the hand to back pocket task calculated via the kinematic model for 3DMC (orange solid line), our deep learning refined kinematic model for Kinect (green dashed line) and the kinematic model for Kinect (blue solid line). The joint angles include shoulder flexion (+)/extension (−), shoulder adduction (+)/abduction (−), shoulder internal rotation (+)/external rotation (−), and elbow flexion (+)/extension (−).

Figure 12

Bland-Altman plots with 95% limits of agreement for joint kinematic parameters during the hand to contralateral shoulder task. X axes represents the angle means of two systems and the Y axes represents the mean of differences. The red line (middle one) represents the reference line at mean, and the two dashed lines represent the upper and lower limit of agreement. The upper four rows are the angles at the point of target achieved (PTA) and the lower four rows are the range of motion (ROM) values. Plots of the left column are measurement differences between our deep learning refined kinematic model $\widehat{\Phi }$ for Kinect and the UWA kinematic model $\Gamma$ for the 3DMC. Plots of the right column are measurement differences between the kinematic model $\Phi$ for Kinect and the UWA kinematic model for the 3DMC $\Gamma$.

Figure 13

Bland-Altman plots with 95% limits of agreement for joint kinematic parameters during the hand to mouth task. X axes represents the angle means of two systems and the Y axes represents the mean of differences. The red line (middle one) represents the reference line at mean, and the two dashed lines represent the upper and lower limit of agreement. The upper four rows are the angles at the point of target achieved (PTA) and the lower four rows are the range of motion (ROM) values. Plots of the left column are measurement differences between our deep learning refined kinematic model $\widehat{\Phi }$ for Kinect and the UWA kinematic model $\Gamma$ for the 3DMC. Plots of the right column are measurement differences between the kinematic model $\Phi$ for Kinect and the UWA kinematic model for the 3DMC $\Gamma$.

Figure 14

Bland-Altman plots with 95% limits of agreement for joint kinematic parameters during the combing hair task. X axes represents the angle means of two systems and the Y axes represents the mean of differences. The red line (middle one) represents the reference line at mean, and the two dashed lines represent the upper and lower limit of agreement. The upper four rows are the angles at the point of target achieved (PTA) and the lower four rows are the range of motion (ROM) values. Plots of the left column are measurement differences between our deep learning refined kinematic model $\widehat{\Phi }$ for Kinect and the UWA kinematic model $\Gamma$ for the 3DMC. Plots of the right column are measurement differences between the kinematic model $\Phi$ for Kinect and the UWA kinematic model for the 3DMC $\Gamma$.

Figure 15

Bland-Altman plots with 95% limits of agreement for joint kinematic parameters during the hand to back pocket task. X axes represents the angle means of two systems and the Y axes represents the mean of differences. The red line (middle one) represents the reference line at mean, and the two dashed lines represent the upper and lower limit of agreement. The upper four rows are the angles at the point of target achieved (PTA) and the lower four rows are the range of motion (ROM) values. Plots of the left column are measurement differences between our deep learning refined kinematic model for Kinect and the UWA kinematic model for the 3DMC. Plots of the left column are measurement differences between our deep learning refined kinematic model $\widehat{\Phi }$ for Kinect and the UWA kinematic model $\Gamma$ for the 3DMC. Plots of the right column are measurement differences between the kinematic model $\Phi$ for Kinect and the UWA kinematic model for the 3DMC $\Gamma$.