A Novel Untethered Hand Wearable with Fine-Grained Cutaneous Haptic Feedback

Abstract

During open surgery, a surgeon relies not only on the detailed view of the organ being operated upon and on being able to feel the fine details of this organ but also heavily relies on the combination of these two senses. In laparoscopic surgery, haptic feedback provides surgeons information on interaction forces between instrument and tissue. There have been many studies to mimic the haptic feedback in laparoscopic-related telerobotics studies to date. However, cutaneous feedback is mostly restricted or limited in haptic feedback-based minimally invasive studies. We argue that fine-grained information is needed in laparoscopic surgeries to study the details of the instrument's end and can convey via cutaneous feedback. We propose an exoskeleton haptic hand wearable which consists of five 4 × 4 miniaturized fingertip actuators, 80 in total, to convey cutaneous feedback. The wearable is described as modular, lightweight, Bluetooth, and WiFi-enabled, and has a maximum power consumption of 830 mW. Software is developed to demonstrate rapid tactile actuation of edges; this allows the user to feel the contours in cutaneous feedback. Moreover, to demonstrate the idea as an object displayed on a flat monitor, initial tests were carried out in 2D. In the second phase, the wearable exoskeleton glove is then further developed to feel 3D virtual objects by using a virtual reality (VR) headset demonstrated by a VR environment. Two-dimensional and 3D objects were tested by our novel untethered haptic hand wearable. Our results show that untethered humans understand actuation in cutaneous feedback just in a single tapping with 92.22% accuracy. Our wearable has an average latency of 46.5 ms, which is much less than the 600 ms tolerable delay acceptable by a surgeon in teleoperation. Therefore, we suggest our untethered hand wearable to enhance multimodal perception in minimally invasive surgeries to naturally feel the immediate environments of the instruments.

Keywords: cutaneous feedback; haptic devices; medical training; teleoperation; virtual reality; wearable devices.

Figure 1

Teleoperation system with ungrounded cutaneous feedback to the operator based on the diagram by Paccheirotti et al. [5]. The cutaneous feedback to the human operator gives information about the forces exerted at the slave side and does not affect the stability of the control loop [5].

Figure 2

Untethered high-resolution haptic hand wearable having 80 tactile actuators (tactors). There is a 4 × 4 matrix of tactors on each fingertip made from Metec P20 Braille cells. The prototype has an open-backhand and open-palm design for easy hand tracking. (a) Open-palm view, (b) open-backhand view, (c) side view of the fingertip tactile matrix with an invisible rubber band to increase the grip of the nail clip if the user has a small or thin finger, and (d) 4 × 4 fingertip tactile matrix made from two P20 Braille cells.

Figure 3

Our initial miniaturized fingertip module presented in 2020, 4 × 4 fingertip tactile matrix actuator with edge detection scanning ROI simulator presented in [55]. (a) Hardware setup, (b) graphical user interface (GUI).

Figure 4

Whole system schematic of the untethered high-resolution haptic hand wearable: (a) power supply block, (b) microcontroller block, and (c) P20 Braille cells block.

Figure 5

P20 Braille cell. (a) The 4 × 4 matrix from two pieces of P20 Braille cells, (b) two-position backplane for two P20 Braille cells, and (c) P20 side-view dimensions [49].

Figure 6

ROI algorithm flow chart.

Figure 7

Fingertip tactile matrices, and graphical user interface (GUI) developed using Processing. The activated pins of (a) index fingertip and (b) thumb tactile matrices correspond to each small section, and (c) corresponding activation of index shown in Figure 7a and thumb in Figure 7b are shown in shaded green in simulator matrices in the GUI.

Figure 8

The integrated experimental setup for 2D scanning.

Figure 9

How a mouse can be a tactile mouse: when the user wears our untethered haptic hand wearable.

Figure 10

An average of 46.5 ms latency was measured from 50 samples.

Figure 11

Edge detection using Canny edge algorithm with Hough transform to thicken the lines. (a) RGB image of a heart, (b) Canny edge result (white lines in black background), (c) inverted Canny edge, (d) thickened Canny edge using Hough transform, and (e) edge detection with tactile matrices simulator.

Figure 12

Tapping vibration. (a) Tapping vibration can be achieved using (i) 1 pin, (ii) 1 row, (iii) 1 column, and (iv) all pins. (b) Different tapping frequencies can be assigned to various shades of gray.

Figure 13

Tapping vs buzzing vibration. (a) Tapping: 5 Hz vibration has clear and distinct vibrations. (b) Buzzing: The low-frequency 5 Hz on–off signal for the vibration motor (in red dotted lines) has a high-frequency buzzing vibrations (in blue spikes). The inherent high-frequency vibrations within low-frequency on-time is due to the rotating mass inside the vibration motor running at 14,000 rpm or around 230 Hz.

Figure 14

Setup for 3D surface scanning of VR objects using Oculus Quest 2 VR headset.

Figure 15

Scaling of the VR tactile actuators.

Figure 16

3D VR environment.

Figure 17

Touching 3D VR objects. (a) Touching the surface of a VR sphere, (b) touching the edge of a cube, and (c) touching the corner of a cube.

Figure 18

Patterns for spatial test.

Figure 19

Patterns for temporal test.

Figure 20

Patterns for 2D scanning test. (a) Experimenter’s screen, (b) participant’s screen. Participants were able to see empty boxes to explore and identify the objects A to I.

Figure 21

2D training patterns.

Figure 22

Spatial training results.

Figure 23

Spatial Test Results.

Figure 24

Temporal training results.

Figure 25

Temporal test results (index only).

Figure 26

2D scanning test confusion matrix.