Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Dec 18;21(24):8469.
doi: 10.3390/s21248469.

Hand Pose Recognition Using Parallel Multi Stream CNN

Iram Noreen  1 , Muhammad Hamid  2 , Uzma Akram  1 , Saadia Malik  3 , Muhammad Saleem  4
Affiliations

Hand Pose Recognition Using Parallel Multi Stream CNN

Iram Noreen et al. Sensors (Basel). .

Abstract

Recently, several computer applications provided operating mode through pointing fingers, waving hands, and with body movement instead of a mouse, keyboard, audio, or touch input such as sign language recognition, robot control, games, appliances control, and smart surveillance. With the increase of hand-pose-based applications, new challenges in this domain have also emerged. Support vector machines and neural networks have been extensively used in this domain using conventional RGB data, which are not very effective for adequate performance. Recently, depth data have become popular due to better understating of posture attributes. In this study, a multiple parallel stream 2D CNN (two-dimensional convolution neural network) model is proposed to recognize the hand postures. The proposed model comprises multiple steps and layers to detect hand poses from image maps obtained from depth data. The hyper parameters of the proposed model are tuned through experimental analysis. Three publicly available benchmark datasets: Kaggle, First Person, and Dexter, are used independently to train and test the proposed approach. The accuracy of the proposed method is 99.99%, 99.48%, and 98% using the Kaggle hand posture dataset, First Person hand posture dataset, and Dexter dataset, respectively. Further, the results obtained for F1 and AUC scores are also near-optimal. Comparative analysis with state-of-the-art shows that the proposed model outperforms the previous methods.

Keywords: 2D CNN; classification; deep learning; depth data; hand posture; multi stream.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Figure 1
Figure 1
A few hand posture samples from selected hand posture datasets, (left) Kaggle Samples [33], (middle) Dexter Samples [22], (right) First Person Samples [34].
Figure 2
Figure 2
Proposed methodology.
Figure 3
Figure 3
Confusion matrices of the performance results by the proposed model by using (a) Kaggle, (b) Dexter, and (c) First Person datasets.
Figure 3
Figure 3
Confusion matrices of the performance results by the proposed model by using (a) Kaggle, (b) Dexter, and (c) First Person datasets.
Figure 4
Figure 4
Macro average ROC curve and AUC score by the proposed approach using the Kaggle dataset.
Figure 5
Figure 5
Macro average ROC curve and AUC score by the proposed approach using the Dexter dataset.
Figure 6
Figure 6
Macro average ROC curve and AUC score by the proposed approach using the First Person dataset.
Figure 7
Figure 7
Plot of accuracy and loss for validation.
See this image and copyright information in PMC

References

    1. Zhu C., Sheng W. Wearable sensor-based hand gesture and daily activity recognition for robot-assisted living. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 2011;41:569–573. doi: 10.1109/TSMCA.2010.2093883. - DOI
    1. Su H., Ovur S.E., Zhou X., Qi W., Ferrigno G., De Momi E. Depth vision guided hand gesture recognition using electromyographic signals. Adv. Robot. 2020;34:985–997. doi: 10.1080/01691864.2020.1713886. - DOI
    1. Van Amsterdam B., Clarkson M.J., Stoyanov D. Gesture Recognition in Robotic Surgery: A Review. IEEE Trans. Biomed. Eng. 2021;68:2021–2035. doi: 10.1109/TBME.2021.3054828. - DOI - PubMed
    1. Danafar S., Gheissari N. Lecture Notes in Computer Science. Volume 4844. Springer; Berlin/Heidelberg, Germany: 2007. Action recognition for surveillance applications using optic flow and SVM; pp. 457–466.
    1. Núñez J.C., Cabido R., Pantrigo J.J., Montemayor A.S., Vélez J.F. Convolutional Neural Networks and Long Short-Term Memory for skeleton-based human activity and hand gesture recognition. Pattern Recognit. 2018;76:80–94. doi: 10.1016/j.patcog.2017.10.033. - DOI

LinkOut - more resources