Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Oct 24;12(1):17812.
doi: 10.1038/s41598-022-21636-z.

Improved 3D-ResNet sign language recognition algorithm with enhanced hand features

Shiqi Wang  1 , Kankan Wang  1 , Tingping Yang  1 , Yiming Li  1 , Di Fan  2
Affiliations

Improved 3D-ResNet sign language recognition algorithm with enhanced hand features

Shiqi Wang et al. Sci Rep. .

Abstract

In sign language video, the hand region is small, the resolution is low, the motion speed is fast, and there are cross occlusion and blur phenomena, which have a great impact on sign language recognition rate and speed, and are important factors restricting sign language recognition performance. To solve these problems, this paper proposes an improved 3D-ResNet sign language recognition algorithm with enhanced hand features, aiming to highlight the features of both hands, solve the problem of missing more effective information when relying only on global features, and improve the accuracy of sign language recognition. The proposed method has two improvements. Firstly, the algorithm detects the left and right hand regions based on the improved EfficientDet network, uses the improved Bi-FPN module and dual channel and spatial attention module are used to enhance the detection ability of the network for small targets like hand. Secondly, the improved residual module is used to improve the 3D-ResNet18 network to extract sign language features. The global, the left-hand and the right-hand image sequences are divided into three branches for feature extraction and fusion, so as to strengthen the attention to hand features, strengthen the representation ability of sign language features, and achieve the purpose of improving the accuracy of sign language recognition. In order to verify the performance of this algorithm, a series of experiments are carried out on CSL dataset. For example, in the experiments of hand detection algorithm and sign language recognition algorithm, the performance indicators such as Top-N, mAP, FLOPs and Parm are applied to find the optimal algorithm framework. The experimental results show that the Top1 recognition accuracy of this algorithm reaches 91.12%, which is more than 10% higher than that of C3D, P3D and 3D-ResNet basic networks. From the performance indicators of Top-N, mAP, FLOPs, Parm and so on, the performance of the algorithm in this paper is better than several algorithms in recent three years, such as I3D+BLSTM, B3D ResNet, AM-ResC3D+RCNN and so on. The results show that the hand detection network with enhanced hand features and three-dimensional convolutional neural network proposed in this paper can achieve higher accuracy of sign language recognition.

PubMed Disclaimer

Conflict of interest statement

The authors declare no competing interests.

Figures

Figure 1
Figure 1
Framework of improved 3D-ResNet sign language recognition algorithm to enhance hand features.
Figure 2
Figure 2
Framework of hand detection algorithm based on EfficientDet and integrating DCSAM attention.
Figure 3
Figure 3
Pseudocode of improved EfficientDet hand detection algorithm.
Figure 4
Figure 4
Convolution structure diagram of EfficientNet.
Figure 5
Figure 5
Bi-FPN module improved in this paper.
Figure 6
Figure 6
Improved enhanced feature extraction module pseudocode.
Figure 7
Figure 7
Designed DCSAM attention module.
Figure 8
Figure 8
DCSAM attention module pseudocode.
Figure 9
Figure 9
Improved 3D-ResNet18 network structure.
Figure 10
Figure 10
Improved 3D-ResNet18 network algorithm pseudocode.
Figure 11
Figure 11
Improved residual structure.
Figure 12
Figure 12
Pseudocode of improved residual network.
Figure 13
Figure 13
Left-hand and right-hand region image sequences.
Figure 14
Figure 14
Pseudocode of three features fusion algorithm.
Figure 15
Figure 15
Schematic diagram of annotation image.
Figure 16
Figure 16
P-R curve comparison diagram, (a) P-R curve of this algorithm under different IOU thresholds, (b) P-R curve of different detection algorithms when IOU = 0.5.
Figure 17
Figure 17
The hand detection effect of this model.
Figure 18
Figure 18
Comparison of recognition accuracy of different algorithms.
See this image and copyright information in PMC

References

    1. Minawaer A, Alifu K, Xie Q, Geng L. Review of sign language recognition methods and techniques. Comput. Eng. Appl. 2021;57:1–12.
    1. Guo D, Tang S, Hong R, Wang M. Review of sign language recognition, translation and generation. Comput. Sci. 2021;48:60–70.
    1. Cheok MJ, Omar Z, Jaward MH. A review of hand gesture and sign language recognition techniques. Int. J. Mach. Learn. Cybern. 2019;10:131–153. doi: 10.1007/s13042-017-0705-5. - DOI
    1. Wu C, et al. Digital gesture recognition method based on data glove and neural networks. J. Southeast Univ. Nat. Sci. Ed. 2020;50:563–569.
    1. Lee S, Choi Y, Sung M, Bae J, Choi Y. A knitted sensing glove for human hand postures pattern recognition. Sensors. 2021;21:1–15. - PMC - PubMed

Publication types