AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove

Abstract

Sign language recognition, especially the sentence recognition, is of great significance for lowering the communication barrier between the hearing/speech impaired and the non-signers. The general glove solutions, which are employed to detect motions of our dexterous hands, only achieve recognizing discrete single gestures (i.e., numbers, letters, or words) instead of sentences, far from satisfying the meet of the signers' daily communication. Here, we propose an artificial intelligence enabled sign language recognition and communication system comprising sensing gloves, deep learning block, and virtual reality interface. Non-segmentation and segmentation assisted deep learning model achieves the recognition of 50 words and 20 sentences. Significantly, the segmentation approach splits entire sentence signals into word units. Then the deep learning model recognizes all word elements and reversely reconstructs and recognizes sentences. Furthermore, new/never-seen sentences created by new-order word elements recombination can be recognized with an average correct rate of 86.67%. Finally, the sign language recognition results are projected into virtual space and translated into text and audio, allowing the remote and bidirectional communication between signers and non-signers.

Fig. 1. The glove configuration and sensor characterization.

a Schematics of the sign language recognition and communication system. b Proportion of different motions that are commonly used in sign language helps determine sensor position on gloves. c Sensor position on gloves based on hand motion analysis in b. Detailed area information of the sensor on each position can be found in Supplementary Fig. 1. d Materials of the triboelectric sensor. e–h Voltage output dependence on key parameters, including sensor area, force, bending degree, and bending speed. The hand, head, and phone images are created by the authors via Blender.

Fig. 2. The data analysis for signals of 50 words and 20 sentences.

a Part of representatives among 50 words or gestures (showing 19 gestures here), in which the opaque and translucent gesture images show the starting and final state of the gesture, respectively. The rest of 31 gesture photo and their corresponding triboelectric signals can be found in Supplementary Fig. 2. b Triboelectric voltage output of 19 words (top), and the similarity and correlation analysis based on the word signals (bottom). The high correlation coefficient of ‘Get’ and ‘Must’ shows a high similarity between these two gesture signals, indicating a high possibility for wrong classification. c Correlation coefficient matrix of signals of 50 words. d Correlation coefficient distribution curve of 50 words. e Correlation coefficient matrix of 20 sign language sentences. f Correlation coefficient distribution curve of 20 sentences. g Voltage output of 20 sentences. Photo credit: Feng Wen, National University of Singapore. Source data of Figs. 2c, e are provided in Harvard Dataverse.

Fig. 3. Word and sentence recognition based on the nonsegmentation method.

a–c Optimization of CNN structure parameters based on accuracy performance, including Kernel size, the number of filters, and the number of convolutional layers. Boxplots indicate median (middle line), 25th, 75th percentile (box) and 5th and 95th percentile (whiskers) as well as outliers (single points). d Final structure of CNN after optimization. e–f Cluster results of word signal from CNN input layer and output layer. g Confusion map of recognizing 50 words. h–i Cluster results of sentence signal from CNN input layer and output layer. j Confusion map of recognizing 17 sentences. Source data of Figs. 3g, j are provided in Harvard Dataverse.

Fig. 4. Word and sentence recognition based on segmentation method, which brings the feasibility of new/never-seen sentence recognition.

a Label table (W01–W20) of 19 words (they are among the total 50 words) that present in 20 sentences (using Y1–Y17 and New1–New3 denote). b Schematic diagram of sentence signal segmentation, ‘The dog scared me’ is taken as an example. c Summary table of sentences with category remarks, comprised words, label series, and unique labeled number order. The same word using the same color to mark. d Schematic diagram of single classifier. e Confusion map of split word element recognition (accuracy 81.9%) based on single classifier. f With successful recognition of each element in sentences, the sentences can be inversely reconstructed and recognized at an average correct rate of 79.41% within single classifier. The dark green means right recognition and light blue means wrong prediction. g Schematic diagram of the hierarchy classifier. h Confusion map of segmented word element recognition (accuracy 82.8%) based on hierarchy classifer. i With successful recognition of each element in sentences, the sentence can be inversely reconstructed and recognition at an average correct rate of 85.58% within hierarchy classifier. j Recognition process of three new sentences that the CNN model did not learn before, taking ‘I lost my dog’ as an example. The detailed recognition results can be found in Supplementary Table 2. Source data of Figs. 4e, h are provided in Harvard Dataverse.

Fig. 5. The demonstration of communication between the speech impaired and the nonsigner.

a Flow chart of the sign language recognition and communication system, which allows the signer to use sign language and nonsigner to types directly to engage in the interaction. The delivered sign language by the signer is recognized and translated into text and speech by AI block. Based on TCP/IP, the client (controlled by signer Lily) in VR interface receives the recognition results and transmits to the sever (operated by the nonsigner Mary). The nonsigner types on the chat box to respond to the signer. b (i–v) Communication/conversation process in VR interface between the speech-disordered user Lily and nonsigner Mary based on the sign language recognition and communication system. The red rectangle indicates the corresponding reaction of these two users. These photos are of one of the authors. c Conversation summary of b. The hand image is created by the authors via Blender. Photo credit: Feng Wen, National University of Singapore.