Computer Vision and Machine Learning-Based Gait Pattern Recognition for Flat Fall Prediction

Abstract

Background: Gait recognition has been applied in the prediction of the probability of elderly flat ground fall, functional evaluation during rehabilitation, and the training of patients with lower extremity motor dysfunction. Gait distinguishing between seemingly similar kinematic patterns associated with different pathological entities is a challenge for the clinician. How to realize automatic identification and judgment of abnormal gait is a significant challenge in clinical practice. The long-term goal of our study is to develop a gait recognition computer vision system using artificial intelligence (AI) and machine learning (ML) computing. This study aims to find an optimal ML algorithm using computer vision techniques and measure variables from lower limbs to classify gait patterns in healthy people. The purpose of this study is to determine the feasibility of computer vision and machine learning (ML) computing in discriminating different gait patterns associated with flat-ground falls.

Methods: We used the Kinect^® Motion system to capture the spatiotemporal gait data from seven healthy subjects in three walking trials, including normal gait, pelvic-obliquity-gait, and knee-hyperextension-gait walking. Four different classification methods including convolutional neural network (CNN), support vector machine (SVM), K-nearest neighbors (KNN), and long short-term memory (LSTM) neural networks were used to automatically classify three gait patterns. Overall, 750 sets of data were collected, and the dataset was divided into 80% for algorithm training and 20% for evaluation.

Results: The SVM and KNN had a higher accuracy than CNN and LSTM. The SVM (94.9 ± 3.36%) had the highest accuracy in the classification of gait patterns, followed by KNN (94.0 ± 4.22%). The accuracy of CNN was 87.6 ± 7.50% and that of LSTM 83.6 ± 5.35%.

Conclusions: This study revealed that the proposed AI machine learning (ML) techniques can be used to design gait biometric systems and machine vision for gait pattern recognition. Potentially, this method can be used to remotely evaluate elderly patients and help clinicians make decisions regarding disposition, follow-up, and treatment.

Keywords: convolutional neural network; fall recognition; gait; k nearest neighbor; long short-time memory; machine learning; pattern recognition; support vector machine.

Figure 1

Illustration of gait kinematics information acquisition, including angles and velocity of the joints using Kinect skeletal tracking SDK, Open CV, and Visual Studio software. Kinematics information of 3 gait patterns was obtained for offline data analysis. (a) The tracking sequence of normal gait (NG); (b) The tracking sequence of pelvic obliquity (PO) gait; (c) The tracking sequence of knee hyperextension (KH) gait.

Figure 2

The human joints that can be obtained by the Kinect SDK camera and the Human Skeleton Recognition and Tracking software toolkits. These 32 joint points are automatically generated by Kinect SDK software based on machine-detected motions. Points A-I represent the joints used in this study for joint angle calculations.

Figure 3

Proposed CNN structure for gait recognition. There are two, 2D-convolutional layers with the following parameters: f = number of filters, k = kernel size, s = stride, p = padding, and n = number of nodes.

Figure 4

The architecture of the proposed LSTM model.

Figure 5

Graph of right lower limb joint angles over time during walking measured by the Human Skeleton Recognition and Tracking software. (a) Hip joint flexion-extension angle; (b) Hip joint abduction-adduction angle; (c) Knee joint angle. NG (normal gait) represents the angles in normal gait, PO (pelvic obliquity) represents the angles during pelvic obliquity gait, and KH (knee hyperextension) represents the angles during knee hyperextension gait. The upper row shows joint angles over time. The lower row demonstrates the measured joint angles (blue lines) in this study.

Figure 6

CNN training loss value over iterations. The classic loss curves were found in the CNN model training and model validation. The loss value of the CNN model converges rapidly in the early stage of training. The loss curve plateaus after about 200 iterations, indicating that the model has converged after 200 iterations.

Figure 7

SVM training loss value over iterations. The process of automatic optimization of the SVM model shows the observation error decreases with the increase in iterations. At the 30th iteration, the SVM model has the smallest loss value, indicating that the SVM model achieves the most optimal parameters.

Figure 8

KNN training loss value over iterations. The KNN model yields the smallest classification error around the 60th iteration, indicating that the KNN model has achieved the optimal parameters.

Figure 9

LSTM training loss value over iterations. There is an overfitting event that occurred over the training iterations. There is no convergence over the training iterations; however, the loss value drops over the iterations.

Figure 10

Comparison of the average classification accuracy of different models.

Figure 11

Confusion matrix result of four classification methods: (a) CNN; (b) LSTM; (c) SVM; (d) KNN.