A multi-modal sensing system for human-robot interaction through tactile and proximity data

Abstract

Introduction: The rapid advancement of collaborative robotics has driven significant interest in Human-Robot Interaction (HRI), particularly in scenarios where robots work alongside humans. This paper considers tasks where a human operator teaches the robot an operation that is then performed autonomously.

Methods: A multi-modal approach employing tactile fingers and proximity sensors is proposed, where tactile fingers serve as an interface, while proximity sensors enable end-effector movements through contactless interactions and collision avoidance algorithms. In addition, the system is modular to make it adaptable to different tasks.

Results: Demonstrative tests show the effectiveness of the proposed system and algorithms. The results illustrate how the tactile and proximity sensors can be used separately or in a combined way to achieve human-robot collaboration.

Discussion: The paper demonstrates the use of the proposed system for tasks involving the manipulation of electrical wires. Further studies will investigate how it behaves with object of different shapes and in more complex tasks.

Keywords: human-robot collaboration; human-robot interaction; modular; multi-modal; proximity sensor; tactile sensor.

FIGURE 1

(a) CAD of the tactile finger showing the mechanical components. (b) Electronic board of the tactile finger. (c) FEA results for the tactile finger. (d) Samples of deformable pad and rigid grid.

FIGURE 2

(a) CAD model of the proximity sensor. (b) Real proximity sensor with 3 ToF modules.

FIGURE 3

(a) Gripper with two pairs of tactile fingers (blue = active; black = passive) and orientation of TCP reference frame. (b) Voltage signals naming convention and reference frame $({\mathrm{\Sigma }}_{sk})$ .

FIGURE 4

Schematic representations of the contact indicator behaviour in different conditions: (a) force applied along $y$ -axis of the ${\mathrm{\Sigma }}_{\mathit{TCP}}$ (top view); (b) moment applied about $z$ -axis of ${\mathrm{\Sigma }}_{\mathit{TCP}}$ (top view); (c) force applied along $z$ -axis of ${\mathrm{\Sigma }}_{\mathit{TCP}}$ (frontal view); (d) moment applied about $x$ -axis of ${\mathrm{\Sigma }}_{\mathit{TCP}}$ (frontal view).

FIGURE 5

Experimental setup with ${\mathrm{\Sigma }}_{\mathit{TCP}}$ frame.

FIGURE 6

(a) Software architecture for the teaching-by-demonstration use case. (b) Flowchart of the teaching-by-demonstration task.

FIGURE 7

(a) Indicator $I_{\mathit{transly}}$ and commanded velocity $u_y$ . (b) Indicator $I_{\mathit{rotx}}$ and commanded velocity ${\omega }_x$ .

FIGURE 8

Indicators and corresponding velocities during the teaching phase.

FIGURE 9

Operator guiding the robot by acting on the grasped electrical wire. From the starting point (a), the operator moves the robot along the $y$ -axis (in green) by pulling the wire in the corresponding direction (b). The robot is then moved downward by pushing the wire down (c) and rotated anti-clockwise about the $z$ -axis (in blue) (d). The robot is then translated again along the $y$ -axis and rotated in the opposite direction (e, f). Finally, the robot end effector is translated and rotated to be positioned above the target clip (g), where the wire is then inserted (h).

FIGURE 10

Indicator $I_{\mathit{transly}}$ during the autonomous wire routing.

FIGURE 11

Measured distance and commanded velocity during the collision avoidance experiment.

FIGURE 12

Collision avoidance involving two proximity sensor modules. The operator simulates obstacles on two sides of the end effector and the robot reacts by moving away from the operator’s hands.

FIGURE 13

Distance measured and commanded velocity during the user following experiment.

FIGURE 14

Hand guidance and obstacle avoidance. An operator guides the robot by pulling the grasped wire while another operator simulates an obstacle nearby the end effector. The robot reacts by following the operator guidance but moving far from the obstacle.

FIGURE 15

Measured distance, indicator, and velocities during the obstacle avoidance in hand guidance experiment.