Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Mar 27;23(7):3509.
doi: 10.3390/s23073509.

Design of Digital-Twin Human-Machine Interface Sensor with Intelligent Finger Gesture Recognition

Dong-Han Mo  1 , Chuen-Lin Tien  1   2 , Yu-Ling Yeh  3 , Yi-Ru Guo  4 , Chern-Sheng Lin  3 , Chih-Chin Chen  5 , Che-Ming Chang  1
Affiliations

Design of Digital-Twin Human-Machine Interface Sensor with Intelligent Finger Gesture Recognition

Dong-Han Mo et al. Sensors (Basel). .

Abstract

In this study, the design of a Digital-twin human-machine interface sensor (DT-HMIS) is proposed. This is a digital-twin sensor (DT-Sensor) that can meet the demands of human-machine automation collaboration in Industry 5.0. The DT-HMIS allows users/patients to add, modify, delete, query, and restore their previously memorized DT finger gesture mapping model and programmable logic controller (PLC) logic program, enabling the operation or access of the programmable controller input-output (I/O) interface and achieving the extended limb collaboration capability of users/patients. The system has two main functions: the first is gesture-encoded virtual manipulation, which indirectly accesses the PLC through the DT mapping model to complete control of electronic peripherals for extension-limbs ability by executing logic control program instructions. The second is gesture-based virtual manipulation to help non-verbal individuals create special verbal sentences through gesture commands to improve their expression ability. The design method uses primitive image processing and eight-way dual-bit signal processing algorithms to capture the movement of human finger gestures and convert them into digital signals. The system service maps control instructions by observing the digital signals of the DT-HMIS and drives motion control through mechatronics integration or speech synthesis feedback to express the operation requirements of inconvenient work or complex handheld physical tools. Based on the human-machine interface sensor of DT computer vision, it can reflect the user's command status without the need for additional wearable devices and promote interaction with the virtual world. When used for patients, the system ensures that the user's virtual control is mapped to physical device control, providing the convenience of independent operation while reducing caregiver fatigue. This study shows that the recognition accuracy can reach 99%, demonstrating practicality and application prospects. In future applications, users/patients can interact virtually with other peripheral devices through the DT-HMIS to meet their own interaction needs and promote industry progress.

Keywords: DT-Sensor; Industry 5.0; PLC; computer vision; digital-twin; digital-twin human-machine interface sensor (DT-HMIS); extension-limbs; finger gesture; mechatronics; virtual manipulation.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Figure 1
Figure 1
The process of transforming DT-HMIS into instruction control commands or reassembling speech.
Figure 2
Figure 2
The DT-HMIS service and processing flow chart.
Figure 3
Figure 3
RGB to the HSV processing results.
Figure 4
Figure 4
Realize gestures and background segmentation.
Figure 5
Figure 5
Improved region-based comparative search method.
Figure 6
Figure 6
The variation scope of the moving objects.
Figure 7
Figure 7
The right diagram shows that a variation area is segmented into small checks.
Figure 8
Figure 8
The numeric DT-MHIS phonetic system. (a) numeric English phonemicized table and (b) frequentation-used sentences.
Figure 9
Figure 9
The PLC corresponds to the X~X4 contacts, where X is the input contact, Y is the output contact, M is the external motor, and COM is the common parallel point. The L and N of the power supply represent the live and neutral wires, respectively. The Q and FX models are PLCs introduced by the MITSUBISHI Company. This PLC, designed for this system, allows the control logic program to be updated through the users such as the DT mapping model in DT-HMIS of this system, without being limited by the developer. Users/patients can also extend peripheral device control on their own.
Figure 10
Figure 10
The flow chart of an experimental system.
Figure 11
Figure 11
Real-world human finger gesture detection in different operating environments. (a) Construction of finger DT model. (b) Measurement of real-world fingers by virtual finger models. (c) Spatial indications of human finger gestures in the real world.
Figure 12
Figure 12
The recognition accuracy of the eight directions in the different backgrounds.
Figure 13
Figure 13
The numeric DT-HMIS using finger gesture recognition (a) eight region (b) screen of the program.
Figure 14
Figure 14
Different finger gesture command parameter identification in various situations: (a) Up, (b) Down, (c) Middle, (d) Right, (e) Upper right, (f) Lower right, (g) Left, (h) Upper left, (i) Lower left, (j) Up, (k) Upper left, (l) Upper right, (m) Left, (n) Upper right.
Figure 14
Figure 14
Different finger gesture command parameter identification in various situations: (a) Up, (b) Down, (c) Middle, (d) Right, (e) Upper right, (f) Lower right, (g) Left, (h) Upper left, (i) Lower left, (j) Up, (k) Upper left, (l) Upper right, (m) Left, (n) Upper right.
Figure 15
Figure 15
The relationship between the stay duration (countdown timer) and accuracy.
Figure 16
Figure 16
The experimental result of operation time for 20 inputs.
See this image and copyright information in PMC

References

    1. Wang H., Lv L., Li X., Li H., Leng J., Zhang Y., Thomson V., Liu G., Wen X., Sun C., et al. A safety management approach for Industry 5.0’s human-centered manufacturing based on digital-twin. J. Manuf. Syst. 2023;66:1–12. doi: 10.1016/j.jmsy.2022.11.013. - DOI
    1. Hultman H., Cedergren S., Wärmefjord K., Söderberg R. Predicting Geometrical Variation in Fabricated Assemblies Using a Digital-twin Approach Including a Novel Non-Nominal Welding Simulation. Aerospace. 2022;9:512. doi: 10.3390/aerospace9090512. - DOI
    1. Han X., Lin Z., Clark C., Vucetic B., Lomax S. AI Based Digital-twin Model for Cattle Caring. Sensors. 2022;22:7118. doi: 10.3390/s22197118. - DOI - PMC - PubMed
    1. Ghandar A., Ahmed A., Zulfiqar S., Hua Z., Hanai M., Theodoropoulos G. A Decision Support System for Urban Agriculture Using Digital-twin: A Case Study with Aquaponics. IEEE Access. 2021;9:35691–35708. doi: 10.1109/ACCESS.2021.3061722. - DOI
    1. Sasikumar A., Vairavasundaram S., Kotecha K., Indragandhi V., Ravi L., Selvachandran G., Abraham A. Blockchain-based trust mechanism for digital-twin empowered Industrial Internet of Things. Future Gener. Comput. Syst. 2023;141:16–27.

LinkOut - more resources