Recognition and Localization of Maize Leaf and Stalk Trajectories in RGB Images Based on Point-Line Net

Abstract

Plant phenotype detection plays a crucial role in understanding and studying plant biology, agriculture, and ecology. It involves the quantification and analysis of various physical traits and characteristics of plants, such as plant height, leaf shape, angle, number, and growth trajectory. By accurately detecting and measuring these phenotypic traits, researchers can gain insights into plant growth, development, stress tolerance, and the influence of environmental factors, which has important implications for crop breeding. Among these phenotypic characteristics, the number of leaves and growth trajectory of the plant are most accessible. Nonetheless, obtaining these phenotypes is labor intensive and financially demanding. With the rapid development of computer vision technology and artificial intelligence, using maize field images to fully analyze plant-related information can greatly eliminate repetitive labor and enhance the efficiency of plant breeding. However, it is still difficult to apply deep learning methods in field environments to determine the number and growth trajectory of leaves and stalks due to the complex backgrounds and serious occlusion problems of crops in field environments. To preliminarily explore the application of deep learning technology to the acquisition of the number of leaves and stalks and the tracking of growth trajectories in field agriculture, in this study, we developed a deep learning method called Point-Line Net, which is based on the Mask R-CNN framework, to automatically recognize maize field RGB images and determine the number and growth trajectory of leaves and stalks. The experimental results demonstrate that the object detection accuracy (mAP50) of our Point-Line Net can reach 81.5%. Moreover, to describe the position and growth of leaves and stalks, we introduced a new lightweight "keypoint" detection branch that achieved a magnitude of 33.5 using our custom distance verification index. Overall, these findings provide valuable insights for future field plant phenotype detection, particularly for datasets with dot and line annotations.

Fig. 1.

Images at 4 angles of the maize dataset with annotations for 3 growth periods: (A) early stage, (B) middle stage, and (C) late stage.

Fig. 2.

Method of determining the ground-truth bounding box. (A) Original annotation. (B) Transformed bounding box annotation. (C) Keypoint annotation after interpolation algorithm processing.

Fig. 3.

Point-Line Net structure based on the Mask R-CNN model: in the box branch, the model inference focuses on the bounding boxes of targets as well as their category, and in the keypoint branch, the model inference focuses on the heatmap of the keypoints in conjunction with the results of the object detection.

Fig. 4.

Feature pyramid network structure.

Fig. 5.

Schematic diagram of the mLD calculation flow: (A) definition of the mLD; (B) the shortest distance from point P to line segment AB is the length of PC; (C) the shortest distance from point P to line segment AB is the length of PB; and (D) the shortest distance from point P to line segment AB is the length of PA.

Fig. 6.

Performance of different object detection models: (A) Precision–recall with different models; (B) mAP50 (%) achieved using different models.

Fig. 7.

Examples of heatmaps generated by both methods: (A) the selected target instance in the example; (B) a single heatmap inferred by Point-Line Net; and (C) heatmaps inferred by the traditional heatmap-based keypoint detection method.

Fig. 8.

Training and validation performance curves: (A) mAP50 (%) curve changes of traditional and innovative methods; (B) loss curve and learning rate curve changes of traditional and innovative methods, where the box loss consists of RPN loss and Fast R-CNN loss [39], and kp loss refers to the cross-entropy loss of the keypoint detection branch; (C) precision–recall; (D) precision–confidence; (E) recall–confidence; and (F) F1–confidence.

Fig. 9.

Prediction results of Point-Line Net in various scenarios: (A) original RGB images; (B) ground truth transformed by the Labelme tool; (C) predicted bounding boxes (the red box represents the identified stem, and the white box represents the identified leaf); (D) predicted keypoints.