Detecting Transitions from Stability to Instability in Robotic Grasping Based on Tactile Perception

Abstract

Robots execute diverse load operations, including carrying, lifting, tilting, and moving objects, involving load changes or transfers. This dynamic process can result in the shift of interactive operations from stability to instability. In this paper, we respond to these dynamic changes by utilizing tactile images captured from tactile sensors during interactions, conducting a study on the dynamic stability and instability in operations, and propose a real-time dynamic state sensing network by integrating convolutional neural networks (CNNs) for spatial feature extraction and long short-term memory (LSTM) networks to capture temporal information. We collect a dataset capturing the entire transition from stable to unstable states during interaction. Employing a sliding window, we sample consecutive frames from the collected dataset and feed them into the network for the state change predictions of robots. The network achieves both real-time temporal sequence prediction at 31.84 ms per inference step and an average classification accuracy of 98.90%. Our experiments demonstrate the network's robustness, maintaining high accuracy even with previously unseen objects.

Keywords: grasp stability prediction; robotic grasping; tactile sensor.

Figure 1

An example of a robot grasping, shifting from a stable grasp to an unstable grasp as the mass of the grasped object increases. In the diagram, the grasping process is divided into two phases: (1) Approaching the object, grasping it, and lifting it; (2) Incrementing the gripper’s load, resulting in the gradual descent of the cup. The shift from a stable to an unstable grasp is visually captured through images recorded by the tactile sensor GelSight.

Figure 2

Description of stable and unstable temporal zone. During the human–robot interaction in Figure 1, when a robot transitions from a stable to an unstable grasping state, the temporal region between the stable state and the vicinity of the stability threshold is referred to as the stable temporal zone. Beyond this threshold, slipping occurs, marking the entry into an unstable state.

Figure 3

Robotic platform. It includes a 6-axis robot arm (JAKA MiniCobo by JAKA Robotics), a two-jaw parallel gripper (PGE-50-26 by DH-Robotics), a RGB-D camera (Intel Realsense D435i; Intel Corporation, Santa Clara, CA, USA), a computer based on LINUX (Ubuntu 20.04.6 LTS), and two tactile sensors (GelSight Mini).

Figure 4

Data collection. Five cups with distinct handles are employed to collect tactile data, incorporating varied grasp forces corresponding to each handle. It is observed that several factors contribute to grasp stability, encompassing the grasping position, applied force, and fluctuations in the object’s weight. Additionally, a notable trend emerged during image acquisition from tactile sensors, revealing rotational occurrences in the detection of object features.

Figure 5

Grasping configuration. We partition the cup’s handle into three sections: upper, middle, and lower. Subsequently, the gripper applies different grasping forces (15 N, 25 N, 40 N, 50 N) to grasp each of the three parts separately.

Figure 6

Overview of network framework. The temporal sequences from the left and right tactile sensors (Gelsight Mini) serve as dual inputs for the classification network. Employing a sliding window of size k frames, we traverse the tactile temporal sequences. Consequently, the input shape of the network is defined as (k, height, width, channel), with k denoting the number of timesteps. The corresponding label for each input is established by determining the maximum label value within the k frames. We utilize pretrained models such as ResNet50 [48], ResNet101 [48], DenseNet121 [49], etc., as the backbone for the network framework. We feed the output of the backbone into sequential models, such as LSTM [54] or Transformer [55], to handle temporal sequences. However, the final choice is contingent upon assessing their classification accuracy, allowing us to determine the most suitable model for our specific application.

Figure 7

m LSTM layers. In the sequential model of Figure 6, the composition of LSTM layers varies. When a single LSTM layer is employed, its output has a shape of [$n\_$feature]. However, if multiple LSTM layers are utilized, the final layer retains the shape [$n\_$feature], while the output of preceding LSTM layers takes the form [k, $n\_$feature]. This is because the outputs of the additional LSTM layers encompass all hidden states across each time step.

Figure 8

Grasp stability prediction on the robotic platform. The two-jaw parallel grippers, equipped with GelSight tactile sensors, randomly grasp the cup’s handle, while the cup’s weight undergoes continuous changes during interactions with humans.