Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 Oct 4;20(19):5670.
doi: 10.3390/s20195670.

Real-Time Fruit Recognition and Grasping Estimation for Robotic Apple Harvesting

Hanwen Kang  1 , Hongyu Zhou  1 , Xing Wang  1 , Chao Chen  1
Affiliations

Real-Time Fruit Recognition and Grasping Estimation for Robotic Apple Harvesting

Hanwen Kang et al. Sensors (Basel). .

Abstract

Robotic harvesting shows a promising aspect in future development of agricultural industry. However, there are many challenges which are still presented in the development of a fully functional robotic harvesting system. Vision is one of the most important keys among these challenges. Traditional vision methods always suffer from defects in accuracy, robustness, and efficiency in real implementation environments. In this work, a fully deep learning-based vision method for autonomous apple harvesting is developed and evaluated. The developed method includes a light-weight one-stage detection and segmentation network for fruit recognition and a PointNet to process the point clouds and estimate a proper approach pose for each fruit before grasping. Fruit recognition network takes raw inputs from RGB-D camera and performs fruit detection and instance segmentation on RGB images. The PointNet grasping network combines depth information and results from the fruit recognition as input and outputs the approach pose of each fruit for robotic arm execution. The developed vision method is evaluated on RGB-D image data which are collected from both laboratory and orchard environments. Robotic harvesting experiments in both indoor and outdoor conditions are also included to validate the performance of the developed harvesting system. Experimental results show that the developed vision method can perform highly efficient and accurate to guide robotic harvesting. Overall, the developed robotic harvesting system achieves 0.8 on harvesting success rate and cycle time is 6.5 seconds.

Keywords: agricultural robot; autonomous harvesting; deep learning; grasping estimation; pointNet; robotic harvesting.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Figure 1
Figure 1
The developed robotic harvesting system that consists of mobile base, manipulator, vision camera, end-effector.
Figure 2
Figure 2
The working principle of the proposed apple harvesting robot.
Figure 3
Figure 3
Network architecture of the Mobile-DasNet, which applies a light-weight backbone and a two-levels FPN to improve the computational efficiency.
Figure 4
Figure 4
The proposed grasping estimation select vector from the fruit centre to surface centre of the visible part as grasp orientation.
Figure 5
Figure 5
Euler-ZYX angle is applied to represent the orientation of the grasp pose.
Figure 6
Figure 6
Network architecture of the PointNet applied in grasping estimation.
Figure 7
Figure 7
Pointset under different conditions, green sphere is the ground truth of the fruit shape.
Figure 8
Figure 8
Fruit recognition and grasping estimation experiments in orchard scenario.
Figure 9
Figure 9
Experiment setup in (a,b) indoor laboratory and (c) outdoor environments.
Figure 10
Figure 10
The process for robotic harvesting experiment in outdoor environment.
Figure 11
Figure 11
Failure grasping estimation in orchard scenario.
See this image and copyright information in PMC

References

    1. Vasconez J.P., Kantor G.A., Cheein F.A.A. Human–robot interaction in agriculture: A survey and current challenges. Biosyst. Eng. 2019;179:35–48. doi: 10.1016/j.biosystemseng.2018.12.005. - DOI
    1. Sa I., Ge Z., Dayoub F., Upcroft B., Perez T., McCool C. Deepfruits: A fruit detection system using deep neural networks. Sensors. 2016;16:1222. doi: 10.3390/s16081222. - DOI - PMC - PubMed
    1. Bac C.W., van Henten E.J., Hemming J., Edan Y. Harvesting robots for high-value crops: State-of-the-art review and challenges ahead. J. Field Robot. 2014;31:888–911. doi: 10.1002/rob.21525. - DOI
    1. Zhao Y., Gong L., Huang Y., Liu C. A review of key techniques of vision-based control for harvesting robot. Comput. Electron. Agric. 2016;127:311–323.
    1. Lin G., Tang Y., Zou X., Xiong J., Li J. Guava detection and pose estimation using a low-cost RGB-D sensor in the field. Sensors. 2019;19:428. doi: 10.3390/s19020428. - DOI - PMC - PubMed

LinkOut - more resources