Guava Detection and Pose Estimation Using a Low-Cost RGB-D Sensor in the Field

Abstract

Fruit detection in real outdoor conditions is necessary for automatic guava harvesting, and the branch-dependent pose of fruits is also crucial to guide a robot to approach and detach the target fruit without colliding with its mother branch. To conduct automatic, collision-free picking, this study investigates a fruit detection and pose estimation method by using a low-cost red⁻green⁻blue⁻depth (RGB-D) sensor. A state-of-the-art fully convolutional network is first deployed to segment the RGB image to output a fruit and branch binary map. Based on the fruit binary map and RGB-D depth image, Euclidean clustering is then applied to group the point cloud into a set of individual fruits. Next, a multiple three-dimensional (3D) line-segments detection method is developed to reconstruct the segmented branches. Finally, the 3D pose of the fruit is estimated using its center position and nearest branch information. A dataset was acquired in an outdoor orchard to evaluate the performance of the proposed method. Quantitative experiments showed that the precision and recall of guava fruit detection were 0.983 and 0.948, respectively; the 3D pose error was 23.43° ± 14.18°; and the execution time per fruit was 0.565 s. The results demonstrate that the developed method can be applied to a guava-harvesting robot.

Keywords: RGB-D sensor; branch reconstruction; fully convolutional network; guava detection; pose estimation.

Figure 1

The guava-harvesting robot and its vision sensing system. (a) Robot system; (b) vision sensing system.

Figure 2

Flow diagram of the developed vision sensing algorithm.

Figure 3

Fully convolutional network (FCN) configuration. The first row uses a deconvolution stride of 32, resulting in a coarse prediction. The second row fuses the outputs from the conv7 layer, the pool3 layer, and the pool4 layer at stride 8, leading to a finer prediction. The deconvolution parameter is defined as ‘(stride) × deconv’.

Figure 4

Segmentation results of the FCN model. (a) An aligned red–green–blue (RGB) image where black pixels represent objects outside the working range of the Kinect V2 sensor; (b) segmentation result where the red parts represent the fruits, and the green parts are the branches.

Figure 5

Fruit detection results. (a) Fruit point cloud extracted from Figure 4b; (b) clustering results, where each cluster is marked with a random color.

Figure 6

Branch reconstruction process. (a) Branch skeletons extracted from Figure 4b; (b) branch point cloud; (c) detected line segments, where each segment is marked with a random color.

Figure 7

Principle of fruit pose estimation. (a) Schematic diagram; (b) three-dimensional (3D) pose estimation result, where the red array represents the fruit pose.

Figure 8

Example showing ground-truth labels. (a) Ground-truth labels for three classes: fruit (blue), branch (red), and background. (b) Ground-truth fruit (blue) and the corresponding mother branch (red).

Figure 9

Examples illustrating unsuccessful detections.

Figure 10

Examples illustrating the fruit poses estimated by the proposed algorithm. The yellow array represents the fruit pose.

Figure 11

Failure examples. The yellow array represents the estimated pose, while the white array is the ground-truth pose.