Diagnosis of early nitrogen, phosphorus and potassium deficiency categories in rice based on multimodal integration and knowledge distillation

Abstract

Rapid, non-destructive, lightweight and accurate diagnosis of early stage nutrient deficiency in rice is essential for both yield and quality. Traditional diagnostic methods often exhibit low efficiency, reduced accuracy, and a lack of timeliness. To address these issues, a diagnostic method for the early detection of nitrogen, phosphorus, and potassium deficiencies in rice, based on multimodal integration and knowledge distillation, is proposed. In this study, the late rice variety 'Huanghuazhan rice' was selected as the experimental subject for field trials. First, leave images of rice plant were captured using a scanner, and some data preprocessing techniques were utilized to extract image samples from the leaf tip areas of the top one leaf, the top two leaf and the top three leaf. Second, the teacher model was obtained through transfer learning, fine-tuning training and model fusion. The custom neural network model was heuristically customized based on the conventional model. The teacher model then performs knowledge distillation on the custom neural network model, resulting in a lightweight model with high accuracy and low memory consumption, which serves as a feature extractor. Finally, the multimodal features were input into LightGBM for training and the rice nutrient deficiency recognition model, S-RiceNet-D-LightGBM (SRDL), was constructed. The experimental results demonstrate that the SRDL model is an efficient, lightweight model characterized by high accuracy and low memory consumption. It achieved an accuracy score of 0.9501, a macro precision score of 0.9501, a macro recall score of 0.9499, and a macro F1 score of 0.9500, outperforming the VGG16, ResNet101, DenseNet169, InceptionNetV3, MobileNetV2, second only to the performance of the ensemble model. The memory footprint is 23.6 MB, which is slightly higher than that of the MobileNetV3S model. This study provides new insights and viable avenues for the practical implementation of a lightweight model designed for the intelligent diagnosis of crop nutrient deficiency.

Keywords: Ensemble learning; Knowledge distillation; Multimodal features; Rice deficiency diagnosis; Transfer learning.

Fig. 1

Complete diagram of each leaf in rice.

Fig. 2

Nitrogen-deficient leaf tips of the top one leaf (a). Nitrogen-deficient leaf tips of the top two leaf (b). Nitrogen-deficient leaf tips of the top three leaf (c). Potassium-deficient leaf tips of the top one leaf (d). Potassium-deficient leaf tips of the top two leaf (e). Potassium-deficient leaf tips of the top three leaf (f). Phosphorus-deficient leaf tips of the top one leaf (g). Phosphorus-deficient leaf tips of the top two leaf (h). Phosphorus-deficient leaf tips of the top three leaf (i). Leaf tip diagram of each leaf under different deficiency conditions in rice.

Fig. 3

Structure of the model for transfer learning and fine-tuning training.

Fig. 4

Structure of the DemoNet_1 model.

Fig. 5

Structure of the DemoNet_2 model.

Fig. 6

Structure of the DemoNet_2-D model.

Fig. 7

Structure of the S-RiceNet-D model.

Fig. 8

Structure of the SRDL model.

Fig. 9

Flow chart of the complete experiment.

Fig. 10

Illustrates the accuracy and loss graphs of the S-RiceNet-D model throughout the training process.

Fig. 11

Depicts the confusion matrix plot of the SRDL model classification results.