Development of a tongue image-based machine learning tool for the diagnosis of gastric cancer: a prospective multicentre clinical cohort study

Abstract

Background: Tongue images (the colour, size and shape of the tongue and the colour, thickness and moisture content of the tongue coating), reflecting the health state of the whole body according to the theory of traditional Chinese medicine (TCM), have been widely used in China for thousands of years. Herein, we investigated the value of tongue images and the tongue coating microbiome in the diagnosis of gastric cancer (GC).

Methods: From May 2020 to January 2021, we simultaneously collected tongue images and tongue coating samples from 328 patients with GC (all newly diagnosed with GC) and 304 non-gastric cancer (NGC) participants in China, and 16 S rDNA was used to characterize the microbiome of the tongue coating samples. Then, artificial intelligence (AI) deep learning models were established to evaluate the value of tongue images and the tongue coating microbiome in the diagnosis of GC. Considering that tongue imaging is more convenient and economical as a diagnostic tool, we further conducted a prospective multicentre clinical study from May 2020 to March 2022 in China and recruited 937 patients with GC and 1911 participants with NGC from 10 centres across China to further evaluate the role of tongue images in the diagnosis of GC. Moreover, we verified this approach in another independent external validation cohort that included 294 patients with GC and 521 participants with NGC from 7 centres. This study is registered at ClinicalTrials.gov, NCT01090362.

Findings: For the first time, we found that both tongue images and the tongue coating microbiome can be used as tools for the diagnosis of GC, and the area under the curve (AUC) value of the tongue image-based diagnostic model was 0.89. The AUC values of the tongue coating microbiome-based model reached 0.94 using genus data and 0.95 using species data. The results of the prospective multicentre clinical study showed that the AUC values of the three tongue image-based models for GCs reached 0.88-0.92 in the internal verification and 0.83-0.88 in the independent external verification, which were significantly superior to the combination of eight blood biomarkers.

Interpretation: Our results suggest that tongue images can be used as a stable method for GC diagnosis and are significantly superior to conventional blood biomarkers. The three kinds of tongue image-based AI deep learning diagnostic models that we developed can be used to adequately distinguish patients with GC from participants with NGC, even early GC and precancerous lesions, such as atrophic gastritis (AG).

Funding: The National Key R&D Program of China (2021YFA0910100), Program of Zhejiang Provincial TCM Sci-tech Plan (2018ZY006), Medical Science and Technology Project of Zhejiang Province (2022KY114, WKJ-ZJ-2104), Zhejiang Provincial Research Center for Upper Gastrointestinal Tract Cancer (JBZX-202006), Natural Science Foundation of Zhejiang Province (HDMY22H160008), Science and Technology Projects of Zhejiang Province (2019C03049), National Natural Science Foundation of China (82074245, 81973634, 82204828), and Chinese Postdoctoral Science Foundation (2022M713203).

Keywords: AFP, alpha fetoprotein; AG, atrophic gastritis; AI, artificial intelligence; APINet, attentive pairwise interaction neural network; AUC, area under the curve; Artificial intelligence; BC, breast cancer; CA, carbohydrate antigen; CEA, carcinoembryonic antigen; CRC, colorectal cancer; DT, decision tree learning; EC, esophageal cancer; GC, gastric cancer; Gastric cancer; HBPC, hepatobiliary pancreatic carcinoma; HC, healthy control; KNN, K-nearest neighbours; LC, lung cancer; NGC, non-gastric cancers; PCoA, principal coordinates analysis; SG, superficial gastritis; SVM, support vector machine; TCM, traditional Chinese medicine; Tongue coating microbiome; Tongue images; Traditional Chinese medicine; TransFG, transformer architecture for fine-grained recognition.

Fig. 1

Both tongue images and the tongue coating microbiome can be used as diagnostic tools for gastric cancer (GC). (A) Flow diagram of the study design. We simultaneously collected tongue images and tongue coating samples from 328 patients with GC and 304 non-gastric cancer (NGC) participants, and 16 S rDNA was used to characterize the microbiome of these tongue coating samples. Then, artificial intelligence (AI) deep learning models were established to evaluate the value of tongue images and the tongue coating microbiome in the diagnosis of GC. Approximately 80% of the cases were used as the training dataset, and approximately 20% of the cases were used as the validation dataset. (B) Principal coordinate analysis (PCoA) showed that there were significant differences in the microbiome between GCs and NGCs. (C) Heatmap of the top 30 species that differed between patients with GC and participants with NGC. (D) The species richness in patients with GC was significantly higher than that in participants with NGC according to Simpson index analysis. (E) The ROC curve and AUC of the internal validation of the DeepLabV3+ model based on the tongue images. (F–G) The ROC curve and AUC of the internal validation of the tongue microbiome evaluation (TME) model based on the tongue coating microbiome at the genus and species levels. (H–I) The distribution of patients with GC and participants with NGC is displayed at the genus and species levels in the internal verification.

Fig. 2

This was a prospective multicentre clinical study. (A–B) Flow diagram of study design: We recruited 937 patients with GC and 1911 participants with NGC from 10 centres to establish 3 tongue image-based artificial intelligence diagnostic models. Approximately 80% of the cases were used as the training dataset, and approximately 20% of the cases were used as the internal validation dataset. In addition, 294 patients with GC and 521 participants with NGC from 7 centres were recruited as the independent external validation dataset to verify the value of the three diagnostic models. (C) The method used to establish model 1 (APINet model) and its process. (D) The method used to establish model 2 (TransFG model) and its process. (E) The method used to establish model 3 (DeepLabV3+ model) and its process.

Fig. 3

The AI diagnostic model based on tongue images performed significantly better than the combined analysis of the levels of 8 tumour indicators in blood in the diagnosis of GC, and combined analysis of tongue images and the levels of 8 tumour indicators in blood can further improve the diagnosis of GC. (A) The ROC curves and AUCs obtained during the internal validation of the three models based on tongue images. (B) The ROC curves and AUCs obtained during the external validation of the three models based on tongue images. (C) The ROC curves and AUCs obtained during the internal validation of the three models (SVM, DT, and KNN) based on the levels of 8 tumour indexes in blood. (D) The ROC curves and AUCs obtained during the external validation of the three models based on the levels of 8 tumour indexes in blood. (E) The ROC curves and AUCs obtained during the internal validation of the two fusion models. (F) The ROC curves and AUCs obtained during the external validation of the two fusion models. (G) The method used to establish the API fusion diagnostic model and its process. (H) The method used to establish the transfusion diagnostic model and its process.

Fig. 4

Tongue image-based models can be used to well distinguish patients with GC from participants with NGC, even those with early-stage GC and precancerous lesions, such as AG, and the diagnostic value of the tongue image-based model for GC has some relationship with tumour size. The ROC curves and AUCs obtained using the APINet model (A), TransFG model (B) and DeepLabV3+ model (C) for GC with different TNM stages. The ROC curves and AUCs obtained using the APINet model (D), TransFG model (E) and DeepLabV3+ model (F) for different participants with NGC (HCs, SG and AG). The ROC curves and AUCs obtained using the APINet model (G), TransFG model (H) and DeepLabV3+ model (I) for HP infection.

Fig. 5

Probability distribution and traceability of the three tongue image models. (A) In the probability distribution performed during the external verification of the three tongue image models, most of the cases were distributed on both sides; that is, there were fewer cases with a vague diagnosis in the middle of 0.41–0.60. (B) Representative tongue images with different probabilities (the intersection of the three models). (C) Output diagram of the APINet model. (D) Traceability of the TransFG model. (E) Traceability of the DeeplabV3+ model.

Fig. 6

The diagnostic value of the tongue image-based diagnostic models for other tumours (EC, HBPC, CRC, LC). (A) The specificity of the tongue-based diagnostic models for GC and other tumours. (B) The ROC and AUC obtained using the APINet model for GC and other tumours. (C) The ROC and AUC obtained using the TransFG model for GC and other tumours. (D) The ROC and AUC obtained using the DeepLabV3+ model for GC and other tumours. CRC, colorectal cancer; EC, oesophageal cancer; HBPC, hepatobiliary pancreatic carcinoma; LC, lung cancer.