A Two-stream Convolutional Network for Musculoskeletal and Neurological Disorders Prediction

Abstract

Musculoskeletal and neurological disorders are the most common causes of walking problems among older people, and they often lead to diminished quality of life. Analyzing walking motion data manually requires trained professionals and the evaluations may not always be objective. To facilitate early diagnosis, recent deep learning-based methods have shown promising results for automated analysis, which can discover patterns that have not been found in traditional machine learning methods. We observe that existing work mostly applies deep learning on individual joint features such as the time series of joint positions. Due to the challenge of discovering inter-joint features such as the distance between feet (i.e. the stride width) from generally smaller-scale medical datasets, these methods usually perform sub-optimally. As a result, we propose a solution that explicitly takes both individual joint features and inter-joint features as input, relieving the system from the need of discovering more complicated features from small data. Due to the distinctive nature of the two types of features, we introduce a two-stream framework, with one stream learning from the time series of joint position and the other from the time series of relative joint displacement. We further develop a mid-layer fusion module to combine the discovered patterns in these two streams for diagnosis, which results in a complementary representation of the data for better prediction performance. We validate our system with a benchmark dataset of 3D skeleton motion that involves 45 patients with musculoskeletal and neurological disorders, and achieve a prediction accuracy of 95.56%, outperforming state-of-the-art methods.

Keywords: Convolutional neural network; Deep learning; Feature fusion; Musculoskeletal disorders; Neurological disorders.

Fig. 1

The optical motion capture system used with the Helen Hayes marker set structure illustrated in: (a) female example, (b) male example

Fig. 2

The sample of a human walking cycle (progressing from left to right)

Fig. 3

The overview of the skeleton structure

Fig. 4

Overview of our proposed two-stream framework

Fig. 5

Modelling the time series of individual joint positions

Fig. 6

Modelling the inter-joint correlations over time

Fig. 7

Confusion matrices of single-stream and the two-stream networks

Fig. 8

Receiver operating characteristic curves for multi-class disorder classification

Fig. 9

Training and testing loss curves of single-stream and two-stream networks

Fig. 10

Proposed fusion network architecture with different CNN and MaxP combinations

Fig. 11

Visualization of the importance of different joints and relative joint displacements with a healthy sample (upper row) and a muscle weakness sample (lower row). The larger size of black joints represent higher importance. The relative joint displacements attention values are aggregated to each joint to show which joints have more interactions with other joints. The importance of relative joint displacements is visualized from a yellow to emerald green scale, with the yellow color representing higher importance