Real-time detection of laryngopharyngeal cancer using an artificial intelligence-assisted system with multimodal data

Abstract

Background: Laryngopharyngeal cancer (LPC) includes laryngeal and hypopharyngeal cancer, whose early diagnosis can significantly improve the prognosis and quality of life of patients. Pathological biopsy of suspicious cancerous tissue under the guidance of laryngoscopy is the gold standard for diagnosing LPC. However, this subjective examination largely depends on the skills and experience of laryngologists, which increases the possibility of missed diagnoses and repeated unnecessary biopsies. We aimed to develop and validate a deep convolutional neural network-based Laryngopharyngeal Artificial Intelligence Diagnostic System (LPAIDS) for real-time automatically identifying LPC in both laryngoscopy white-light imaging (WLI) and narrow-band imaging (NBI) images to improve the diagnostic accuracy of LPC by reducing diagnostic variation among on-expert laryngologists.

Methods: All 31,543 laryngoscopic images from 2382 patients were categorised into training, verification, and test sets to develop, validate, and internal test LPAIDS. Another 25,063 images from five other hospitals were used as external tests. Overall, 551 videos were used to evaluate the real-time performance of the system, and 200 randomly selected videos were used to compare the diagnostic performance of the LPAIDS with that of laryngologists. Two deep-learning models using either WLI (model W) or NBI (model N) images were constructed to compare with LPAIDS.

Results: LPAIDS had a higher diagnostic performance than models W and N, with accuracies of 0·956 and 0·949 in the internal image and video tests, respectively. The robustness and stability of LPAIDS were validated in external sets with the area under the receiver operating characteristic curve values of 0·965-0·987. In the laryngologist-machine competition, LPAIDS achieved an accuracy of 0·940, which was comparable to expert laryngologists and outperformed other laryngologists with varying qualifications.

Conclusions: LPAIDS provided high accuracy and stability in detecting LPC in real-time, which showed great potential for using LPAIDS to improve the diagnostic accuracy of LPC by reducing diagnostic variation among on-expert laryngologists.

Keywords: Deep-learning models; Diagnostic; Head and neck tumour; Laryngoscopic; Multicentre; Real-time.

Fig. 1

Flowchart for development and evaluation of the LPAIDS for laryngopharyngeal cancer diagnosis. LPAIDS: Laryngopharyngeal Artificial Intelligence Diagnostic System

Fig. 2

Workflow and architecture of LPAIDS. a Procedure for detecting LPC from laryngoscopy videos. The WLI and NBI laryngoscopy video frames were extracted from laryngoscopy videos. After screening and annotation by highly experienced laryngoscopists, the images were fed into the model to localize the area with possible tumours; the diagnoses were based on the shape and size of the tumour area. Three pre-trained convolutional neural network models (model W, model N, and LPAIDS, based on U-Net) were developed to obtain the feature vectors from the WLI, NBI, and all images, respectively. b The detailed neural network architecture of LPAIDS based on U-Net. LPAIDS: Laryngopharyngeal Artificial Intelligence Diagnostic System; LPC: laryngopharyngeal cancer; NBI: narrow-band imaging; WLI: white-light imaging

Fig. 3

Performance of LPAIDS for identifying laryngopharyngeal cancer in the internal image and video datasets. a ROC curves of LPAIDS using all images in the internal image test set. b ROC curves of LPAIDS and model W using WLI images in the internal image test set. c ROC curves of LPAIDS and model N using NBI images in the internal image testing set. d ROC curves of LPAIDS using all videos in the internal video test sets. e ROC curves of LPAIDS and model W using WLI videos in the internal video test sets. f ROC curves of LPAIDS and model N using NBI videos in the internal video test sets. LPAIDS: Laryngopharyngeal Artificial Intelligence Diagnostic System; ROC: receiver operating characteristic; NBI: narrow-band imaging; WLI: white-light imaging

Fig. 4

Representative attention maps obtained by LPAIDS for identifying laryngopharyngeal cancer. The attention map is shown as a heatmap superimposed on the original image, where warmer colors indicate higher saliency. LPAIDS: Laryngopharyngeal Artificial Intelligence Diagnostic System

Fig. 5

The accuracy of LPAIDS in segmentation of laryngopharyngeal cancer regions. a The distribution of IOU for the internal image test sets. b Representative prediction results correspond to various segmentation performances of LPAIDS for laryngopharyngeal cancer segmentation. The green line was labeled by the laryngoscopists, and the red line was labeled by LPAIDS automatic calculation. LPAIDS: Laryngopharyngeal Artificial Intelligence Diagnostic System; IOU: Intersection-Over-Union

Fig. 6

ROC curves illustrating the performance of LPAIDS for identifying laryngopharyngeal cancer in multicentre imaging datasets. FAHSU: First Affiliated Hospital of Shenzhen University; LPAIDS: FAHSYSU: First Affiliated Hospital of Sun Yat-sen University; Laryngopharyngeal Artificial Intelligence Diagnostic System; NHSMU: Nanfang Hospital of Southern Medical University; ROC: receiver operating characteristic; SAHSYSU: Sixth Affiliated Hospital of Sun Yat-sen University; SYMSYSU: Sun Yat-sen Memorial Hospital of Sun Yat-sen University; TAHSYSU: Third Affiliated Hospital of Sun Yat-sen University

Fig. 7

Diagnostic performance for identifying laryngopharyngeal cancer between the LPAIDS and laryngologists in 200 videos. a Receiver operating characteristic curves of LPAIDS, expert, senior, laryngologist residents, and trainees for comparison of the diagnostic performance. b Confusion matrices obtained by LPAIDS and ten laryngologists with varying degrees of expertise. Expert: a professor with > 20 years of experience in endoscopic procedures. Senior: attending doctors with more than five years of experience who had completed clinical and specific endoscopic training. Residents: residents with more than three years of endoscopic experience. Trainee: internsone year of endoscopic experience. LPAIDS: Laryngopharyngeal Artificial Intelligence Diagnostic System