In-Field Automatic Identification of Pomegranates Using a Farmer Robot

Abstract

Ground vehicles equipped with vision-based perception systems can provide a rich source of information for precision agriculture tasks in orchards, including fruit detection and counting, phenotyping, plant growth and health monitoring. This paper presents a semi-supervised deep learning framework for automatic pomegranate detection using a farmer robot equipped with a consumer-grade camera. In contrast to standard deep-learning methods that require time-consuming and labor-intensive image labeling, the proposed system relies on a novel multi-stage transfer learning approach, whereby a pre-trained network is fine-tuned for the target task using images of fruits in controlled conditions, and then it is progressively extended to more complex scenarios towards accurate and efficient segmentation of field images. Results of experimental tests, performed in a commercial pomegranate orchard in southern Italy, are presented using the DeepLabv3+ (Resnet18) architecture, and they are compared with those that were obtained based on conventional manual image annotation. The proposed framework allows for accurate segmentation results, achieving an F1-score of 86.42% and IoU of 97.94%, while relieving the burden of manual labeling.

Keywords: agricultural robotics; deep learning; fruit detection; multi-stage transfer learning; precision farming.

Figure 1

The farmer robot, Polibot, used for experimentation, equipped with a multi-sensor suite.

Figure 2

Sample images from the different data sets: controlled environment acquisitions under uniform (left) and non-uniform (center) exposure; field images acquired by the robotic platform (right).

Figure 3

Pipeline of the image segmentation approach.

Figure 4

Block diagram of the DeepLabv3+ architecture using a set of 18 consecutive convolutional layers.

Figure 5

Sample image from SET1 (left) and the corresponding labeled image obtained by color threshold and morphological operations (right).

Figure 6

Data augmentation for the sample image of Figure 5. Twenty images are obtained by rotating, reflecting and varying contrast and exposure.

Figure 7

Four segmented images of SET2, in which each pixel recognized as a fruit by the First CE net is blue, while those in the background are cyan.

Figure 8

Results of labeling for a sample image of SET2: original image (left); labeled image obtained using the First CE net before (center) andafter morphological operations (right).

Figure 9

Final CE net result for a sample image acquired in the field. (Left): original image; (right): segmentation result.

Figure 10

True negative images acquired in a vineyard used for TNA training.

Figure 11

TNA result for the same image as Figure 9. (Left): original image; (right): segmentation result.

Figure 12

FIA result for the same image as Figure 9. (Left): original image; (right): segmentation result.

Figure 13

Comparison among segmentation results obtained from the three different approaches. White pixels represent true positives, black pixels represent true negatives, pink pixels represent false negatives and green pixels represent false positives.

Figure 14

Confusion matrices for the manual labeling network (left) and the FIA network (right).