Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Oct 8;8(6):479.
doi: 10.3390/biomimetics8060479.

Human Operation Augmentation through Wearable Robotic Limb Integrated with Mixed Reality Device

Hongwei Jing  1 , Tianjiao Zheng  1 , Qinghua Zhang  1 , Kerui Sun  1 , Lele Li  1 , Mingzhu Lai  2 , Jie Zhao  1 , Yanhe Zhu  1
Affiliations

Human Operation Augmentation through Wearable Robotic Limb Integrated with Mixed Reality Device

Hongwei Jing et al. Biomimetics (Basel). .

Abstract

Mixed reality technology can give humans an intuitive visual experience, and combined with the multi-source information of the human body, it can provide a comfortable human-robot interaction experience. This paper applies a mixed reality device (Hololens2) to provide interactive communication between the wearer and the wearable robotic limb (supernumerary robotic limb, SRL). Hololens2 can obtain human body information, including eye gaze, hand gestures, voice input, etc. It can also provide feedback information to the wearer through augmented reality and audio output, which is the communication bridge needed in human-robot interaction. Implementing a wearable robotic arm integrated with HoloLens2 is proposed to augment the wearer's capabilities. Taking two typical practical tasks of cable installation and electrical connector soldering in aircraft manufacturing as examples, the task models and interaction scheme are designed. Finally, human augmentation is evaluated in terms of task completion time statistics.

Keywords: human augmentation; human–robot interaction; mixed reality; supernumerary robotic limb; wearable robotic limb.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Figure 1
Figure 1
SRL system using HoloLens2 as the communication medium. The purple line represents the information flow from the human to SRLs, and the blue line represents the information flow from SRLs to the human.
Figure 2
Figure 2
A diagram of the SRL system integrated with HoloLens. It also shows the application effect of some functions in HoloLens.
Figure 3
Figure 3
HoloLens2 program construction and integration.
Figure 4
Figure 4
SRL interaction framework diagram based on mixed reality and human multimodal information. The blue box contains multimodal information about the human, and the purple box contains mixed reality content. The two are combined to obtain the three kinds of intent information in the orange box. The figure also shows several cases combining human body multimodal information and Mixed Reality.
Figure 5
Figure 5
The coordinate system of HoloLens virtual space, world and the SRL real space.
Figure 6
Figure 6
Description of the state transitions for the cable installation task. The view is the first-person perspective in HoloLens. The whole process is completed in the virtual environment, and the yellow cylinder represents the virtual cable with green installation marking points on it.
Figure 7
Figure 7
Description of state transitions for electrical connector soldering tasks. Red cylinders represent virtual cables. Electrical connectors are fixed at the end of the SRL model. The figure simulates the process of SRL assisting the operator to complete the welding task. When the electrical connector is successfully welded to the cable, the cable turns green.
Figure 8
Figure 8
Schematic diagram of human hand joint information.
Figure 9
Figure 9
Application of Vuforia AR to realize position and posture tracking of the soldering iron.
Figure 10
Figure 10
Organization Chart for Augmentation Effect Evaluation.
Figure 11
Figure 11
Procedure for cable installation tasks. Step 1: The operator organizes the cables. Step 2: The SRL grips and tensions the cable. Step 3: The cables are installed. Step 4: The mission ends, and the SRL returns to the preset position.
Figure 12
Figure 12
Experimental operating procedure for soldering tasks. Step 1: The SRL grabs the electrical connector and enters dragged motion mode. Step 2: The operator drags the end of the SRL to a suitable position. Step 3: The SRL end follows the soldering iron to adjust the angle. Step 4: The operator completes the soldering task. Step 5: The soldering task is completed, and the SRL returns to the preset position.
Figure 13
Figure 13
Mean task time of subjects under different experiment conditions. The figure also shows the percentage reduction in task time. (a) Mean task time for participant a. (b) Mean task time for participant b. (c) Mean task time for participant c. (d) Mean task time for participant d. (e) Mean task time for participant e. (f) Mean task time for participant f. (g) Mean task time for participant g. (h) Mean task time for participant h. (i) Mean task time for participant i. (j) Mean task time for participant j.
Figure 13
Figure 13
Mean task time of subjects under different experiment conditions. The figure also shows the percentage reduction in task time. (a) Mean task time for participant a. (b) Mean task time for participant b. (c) Mean task time for participant c. (d) Mean task time for participant d. (e) Mean task time for participant e. (f) Mean task time for participant f. (g) Mean task time for participant g. (h) Mean task time for participant h. (i) Mean task time for participant i. (j) Mean task time for participant j.
See this image and copyright information in PMC

References

    1. Jing H., Zhu Y., Zhao S., Zhang Q., Zhao J. Research status and development trend of supernumerary robotic limbs. J. Mech. Eng. 2020;26:1–9.
    1. Tong Y., Liu J. Review of research and development of supernumerary robotic limbs. IEEE/CAA J. Autom. Sin. 2021;8:929–952.
    1. Yang B., Huang J., Chen X., Xiong C., Hasegawa Y. Supernumerary robotic limbs: A review and future outlook. IEEE Trans. Med Robot. Bionics. 2021;3:623–639. doi: 10.1109/TMRB.2021.3086016. - DOI
    1. Parietti F., Asada H.H. Supernumerary robotic limbs for aircraft fuselage assembly: Body stabilization and guidance by bracing; Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA); Hong Kong, China. 31 May–7 June 2014; pp. 1176–1183.
    1. Véronneau C., Denis J., Lebel L.P., Denninger M., Blanchard V., Girard A., Plante J.S. Multifunctional remotely actuated 3-DOF supernumerary robotic arm based on magnetorheological clutches and hydrostatic transmission lines. IEEE Robot. Autom. Lett. 2020;5:2546–2553. doi: 10.1109/LRA.2020.2967327. - DOI

LinkOut - more resources