Rapid identification of medicinal plants via visual feature-based deep learning

Abstract

Background: Traditional Chinese Medicinal Plants (CMPs) hold a significant and core status for the healthcare system and cultural heritage in China. It has been practiced and refined with a history of exceeding thousands of years for health-protective affection and clinical treatment in China. It plays an indispensable role in the traditional health landscape and modern medical care. It is important to accurately identify CMPs for avoiding the affected clinical safety and medication efficacy by the different processed conditions and cultivation environment confusion.

Results: In this study, we utilize a self-developed device to obtain high-resolution data. Furthermore, we constructed a visual multi-varieties CMPs image dataset. Firstly, a random local data enhancement preprocessing method is proposed to enrich the feature representation for imbalanced data by random cropping and random shadowing. Then, a novel hybrid supervised pre-training network is proposed to expand the integration of global features within Masked Autoencoders (MAE) by incorporating a parallel classification branch. It can effectively enhance the feature capture capabilities by integrating global features and local details. Besides, the newly designed losses are proposed to strengthen the training efficiency and improve the learning capacity, based on reconstruction loss and classification loss.

Conclusions: Extensive experiments are performed on our dataset as well as the public dataset. Experimental results demonstrate that our method achieves the best performance among the state-of-the-art methods, highlighting the advantages of efficient implementation of plant technology and having good prospects for real-world applications.

Keywords: Deep learning; Identification; Image recognition; Masked autoencoders; Medicinal plants.

Fig. 1

The image detection to detection results. (A) is the image acquisition device. The device is composed of a box, a light system, and an image acquisition system, which can provide stable and consistent environmental conditions. (B) is the obtained medicinal plant images of different types. (C) is the detected images with bounding boxes

Fig. 2

The dataset consists of 14 different CHMs and their produced products. Namely (A) chaoshanzha (B) jiaoshanzha (C) shanzhatan (D) jiangbanxia (E) lubei (F) qingbei (G) songbei (H) fabanxia (I) shengbanxia (J) jingbanxia (K) shuibanxia (L) jiangnanxing (M) shanzha (N) qingbanxia

Fig. 3

The distribution of the number of images within each CMP in our dataset. The blue represents the raw samples, while the orange is the collected original data

Fig. 4

The overview of our identification model

Fig. 5

The Grad-CAM heatmap is based on MAE. The first row and Third row display original images, while the second row and 4-th row show the Grad-CAM heatmap results. The heatmaps are where the model is focused on

Fig. 6

In the processing of Random shadow enhancement, is a random value between 0 to 1, is the added shadow probability

Fig. 7

In the partial results of data augmentation results, each row shows the randomly cropped data of different classes, namely shanzha, qingbanxia, jingbanxia, and jiangbanxia, respectively

Fig. 8

The experimental results of the confusion matrix. The numbers from 0 to 13 correspond to different classes. The columns represent the predicted labels, the rows represent the true labels. The values corresponding to rows and columns have indicated the number of correct classes predicted from true data

Fig. 9

The experimental results of Receiver Operating Characteristic (ROC). The number from 0 to 13 corresponds to different classes. Based on the confusion matrix, ROC is computed to reflect the difference between the True Positive Rate and False Positive Rate. The range of ROC curve is between 0 and 1 (1 is best, 0 is lowest)

Fig. 10

The experimental results of a confusion matrix for different models. (A) VGG (B) CoAtNet (C) DenseNet (D) EffcientNet (E) MobileNets (F) ResNet (G) ViT (H) MAE

Fig. 11

The visualization of the different models for original data. The highlighted areas of the CAM heatmap represent the model considered most relevant to each class. The heat maps of each class are randomly selected. The first is the original image, the second is the no-pretrained MAE, the third is the pretrained MAE, and the last is ours

Fig. 12

The visualization of the different models for different color backgrounds. The heat maps of each class are randomly selected

Fig. 13

The visualization of the different models for different lighting and shadowing. The heat maps of each class are randomly selected

Fig. 14

The visualization of the different models for different reflectance. The heat maps of each class are randomly selected

Fig. 15

The experimental results of different iterations

Fig. 16

The comparison of experimental results of different iterations