PanicleNeRF: Low-Cost, High-Precision In-Field Phenotyping of Rice Panicles with Smartphone

Abstract

The rice panicle traits substantially influence grain yield, making them a primary target for rice phenotyping studies. However, most existing techniques are limited to controlled indoor environments and have difficulty in capturing the rice panicle traits under natural growth conditions. Here, we developed PanicleNeRF, a novel method that enables high-precision and low-cost reconstruction of rice panicle three-dimensional (3D) models in the field based on the video acquired by the smartphone. The proposed method combined the large model Segment Anything Model (SAM) and the small model You Only Look Once version 8 (YOLOv8) to achieve high-precision segmentation of rice panicle images. The neural radiance fields (NeRF) technique was then employed for 3D reconstruction using the images with 2D segmentation. Finally, the resulting point clouds are processed to successfully extract panicle traits. The results show that PanicleNeRF effectively addressed the 2D image segmentation task, achieving a mean F1 score of 86.9% and a mean Intersection over Union (IoU) of 79.8%, with nearly double the boundary overlap (BO) performance compared to YOLOv8. As for point cloud quality, PanicleNeRF significantly outperformed traditional SfM-MVS (structure-from-motion and multi-view stereo) methods, such as COLMAP and Metashape. The panicle length was then accurately extracted with the rRMSE of 2.94% for indica and 1.75% for japonica rice. The panicle volume estimated from 3D point clouds strongly correlated with the grain number (R ² = 0.85 for indica and 0.82 for japonica) and grain mass (0.80 for indica and 0.76 for japonica). This method provides a low-cost solution for high-throughput in-field phenotyping of rice panicles, accelerating the efficiency of rice breeding.

Fig. 1.

Experimental fields and data acquisition. (A) Ripening stage of indica rice at the field of Longping High-Tech in Lingshui, Hainan Province, China. (B) Ripening stage of japonica rice at the field of Jiaxing Academy of Agricultural Sciences in Jiaxing, Zhejiang Province, China. (C) Data acquisition by circling around the target rice panicle using a smartphone.

Fig. 2.

Flowchart of PanicleNeRF method. (A) Data acquisition and preprocessing. (B) 2D image segmentation. (C) 3D reconstruction.

Fig. 3.

The boundary overlap performance of different methods on rice varieties. (A) Performance on indica rice dataset. (B) Performance on japonica rice dataset.

Fig. 4.

Illustration of the representative image segmentation results on indica rice (left column) and japonica rice (right column) by different methods. (A) Mask-RCNN segmentation results. (B) YOLOv8 segmentation results. (C) PanicleNeRF segmentation results.

Fig. 5.

NeRF models of indica and japonica rice reconstructed using original image set and 2D segmented images, viewed from three different perspectives of the NeRF model. (A) Indica NeRF model obtained from the original image set. (B) Indica NeRF model obtained from images with 2D segmentation. (C) Japonica NeRF model obtained from the original image set. (D) Japonica NeRF model obtained from images with 2D segmentation.

Fig. 6.

Illustration of the representative clustering results of rice panicle and label point clouds for (A) indica rice and (B) japonica rice, where blue indicates the panicle semantics and red indicates the label semantics.

Fig. 7.

The comparison of representative rice panicle point clouds reconstructed by PanicleNeRF and traditional 3D reconstruction methods (COLMAP and Metashape). (A) Front view of indica rice. (B) Side view of indica rice. (C) Front view of japonica rice. (D) Side view of japonica rice.

Fig. 8.

Correlation analysis between predicted and measured panicle lengths for (A) indica rice and (B) japonica rice.

Fig. 9.

Correlation analysis between predicted panicle volume and measured grain count and grain mass for indica and japonica rice. (A) Predicted volume versus measured grain count for indica rice. (B) Predicted volume versus measured grain count for japonica rice. (C) Predicted volume versus measured grain mass for indica rice. (D) Predicted volume versus measured grain mass for japonica rice.

Fig. 10.

Heatmap of correlations among panicle length, panicle volume, grain count, and grain mass for different rice panicle types in the indica and japonica datasets.