Learning-Based Slip Detection for Robotic Fruit Grasping and Manipulation under Leaf Interference

Abstract

Robotic harvesting research has seen significant achievements in the past decade, with breakthroughs being made in machine vision, robot manipulation, autonomous navigation and mapping. However, the missing capability of obstacle handling during the grasping process has severely reduced harvest success rate and limited the overall performance of robotic harvesting. This work focuses on leaf interference caused slip detection and handling, where solutions to robotic grasping in an unstructured environment are proposed. Through analysis of the motion and force of fruit grasping under leaf interference, the connection between object slip caused by leaf interference and inadequate harvest performance is identified for the first time in the literature. A learning-based perception and manipulation method is proposed to detect slip that causes problematic grasps of objects, allowing the robot to implement timely reaction. Our results indicate that the proposed algorithm detects grasp slip with an accuracy of 94%. The proposed sensing-based manipulation demonstrated great potential in robotic fruit harvesting, and could be extended to other pick-place applications.

Keywords: leaf interference; long-short-term memory (LSTM); robotic harvesting; slip detection.

Figure 1

Contact and force analysis of typical grasp scenarios. (a) vertical grasp, (b) vertical grasp with thin obstacle, (c) horizontal grasp, and (d) horizontal grasp with leaf interference. Normal forces, equivalent pulling force, and other forces were represented in blue, green, and black color, respectively.

Figure 2

System architecture. (a) data collection; (b) data processing; (c) control and manipulation; (d) tactile perception (tactile data collected in four fingers will be represented with red, green, blue and gray colors in the visualisation, larger force applied on the sensing area will generate darker shade of the color).

Figure 3

Gripper with independent motor-controlled fingers.

Figure 4

Control flow diagram of the proposed closed-loop grasp manipulation.

Figure 5

Grasp force tuning of the designed grasp manipulation.

Figure 6

Proposed slip detection neural network architecture.

Figure 7

Data collection. (a) Gripper open; (b) Gripper close; (c) Leaf sliding directions.

Figure 8

Performance of the proposed LSTM network.(a) Confusion matrix; (b) Epoch-Loss graph; (c) Epoch-Accuracy graph.

Figure 9

Confusion matrices for the proposed LSTM networks with different layers and time steps. (darker color represents higher accuracy.)

Figure 10

Experiment setting of grasp manipulation.

Figure 11

Slip detection and reaction motion sequence. (a) gripper arrived; (b) gripper closed; (c) gripper rotated; (d) slip detected; (e) motor-actuated finger reaction; (f) maximum grasping force applied.