Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Aug 19;24(16):5351.
doi: 10.3390/s24165351.

Bengali-Sign: A Machine Learning-Based Bengali Sign Language Interpretation for Deaf and Non-Verbal People

Md Johir Raihan  1 , Mainul Islam Labib  1 , Abdullah Al Jaid Jim  1   2 , Jun Jiat Tiang  3 , Uzzal Biswas  1 , Abdullah-Al Nahid  1
Affiliations

Bengali-Sign: A Machine Learning-Based Bengali Sign Language Interpretation for Deaf and Non-Verbal People

Md Johir Raihan et al. Sensors (Basel). .

Abstract

Sign language is undoubtedly a common way of communication among deaf and non-verbal people. But it is not common among hearing people to use sign language to express feelings or share information in everyday life. Therefore, a significant communication gap exists between deaf and hearing individuals, despite both groups experiencing similar emotions and sentiments. In this paper, we developed a convolutional neural network-squeeze excitation network to predict the sign language signs and developed a smartphone application to provide access to the ML model to use it. The SE block provides attention to the channel of the image, thus improving the performance of the model. On the other hand, the smartphone application brings the ML model close to people so that everyone can benefit from it. In addition, we used the Shapley additive explanation to interpret the black box nature of the ML model and understand the models working from within. Using our ML model, we achieved an accuracy of 99.86% on the KU-BdSL dataset. The SHAP analysis shows that the model primarily relies on hand-related visual cues to predict sign language signs, aligning with human communication patterns.

Keywords: Bengali sign language (BdSL); SHAP; convolutional neural network (CNN); squeeze excitation (SE).

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflicts of interest.

Figures

Figure 1
Figure 1
Workflow of the proposed sign language prediction framework.
Figure 2
Figure 2
Thirty unique samples from the KU-BdSL dataset.
Figure 3
Figure 3
Data augmentation process of a random sample from the KU-BdSL dataset.
Figure 4
Figure 4
CNN architecture with SE block.
Figure 5
Figure 5
All the functionalities of the proposed smartphone application for sign language prediction.
Figure 6
Figure 6
Results on the testing set.
Figure 7
Figure 7
Confusion matrix of the model on the test set.
Figure 8
Figure 8
Interpreting the CNN model using SHAP.
Figure 9
Figure 9
Developed smartphone application.
Figure 10
Figure 10
Predicated sign of two users.
Figure 11
Figure 11
Predicted outcome of all the samples of User 1.
Figure 12
Figure 12
Predicted outcome of all the samples of User 2.
See this image and copyright information in PMC

References

    1. Tarafder K., Akhtar N., Zaman M., Rasel M., Bhuiyan M., Datta P. Disabling hearing impairment in the Bangladeshi population. J. Laryngol. Otol. 2015;129:126–135. doi: 10.1017/S002221511400348X. - DOI - PMC - PubMed
    1. Hossen M., Govindaiah A., Sultana S., Bhuiyan A. Bengali sign language recognition using deep convolutional neural network; Proceedings of the 2018 Joint 7th International Conference on Informatics, Electronics & Vision (ICIEV) and 2018 2nd International Conference on Imaging, Vision & Pattern Recognition (icIVPR); Kitakyushu, Japan. 25–29 June 2018; pp. 369–373.
    1. Hasan M.M., Srizon A.Y., Hasan M.A.M. Classification of Bengali sign language characters by applying a novel deep convolutional neural network; Proceedings of the 2020 IEEE Region 10 Symposium (TENSYMP); Dhaka, Bangladesh. 5–7 June 2020; pp. 1303–1306.
    1. Hasan M., Sajib T.H., Dey M. A machine learning based approach for the detection and recognition of Bangla sign language; Proceedings of the 2016 International Conference on Medical Engineering, Health Informatics and Technology (MediTec); Dhaka, Bangladesh. 17–18 December 2016; pp. 1–5.
    1. Miah A.S.M., Shin J., Hasan M.A.M., Rahim M.A. Bensignnet: Bengali sign language alphabet recognition using concatenated segmentation and convolutional neural network. Appl. Sci. 2022;12:3933. doi: 10.3390/app12083933. - DOI

LinkOut - more resources