Cervical lymph node metastasis prediction from papillary thyroid carcinoma US videos: a prospective multicenter study

Abstract

Background: Prediction of lymph node metastasis (LNM) is critical for individualized management of papillary thyroid carcinoma (PTC) patients to avoid unnecessary overtreatment as well as undesired under-treatment. Artificial intelligence (AI) trained by thyroid ultrasound (US) may improve prediction performance.

Methods: From September 2017 to December 2018, patients with suspicious PTC from the first medical center of the Chinese PLA general hospital were retrospectively enrolled to pre-train the multi-scale, multi-frame, and dual-direction deep learning (MMD-DL) model. From January 2019 to July 2021, PTC patients from four different centers were prospectively enrolled to fine-tune and independently validate MMD-DL. Its diagnostic performance and auxiliary effect on radiologists were analyzed in terms of receiver operating characteristic (ROC) curves, areas under the ROC curve (AUC), accuracy, sensitivity, and specificity.

Results: In total, 488 PTC patients were enrolled in the pre-training cohort, and 218 PTC patients were included for model fine-tuning (n = 109), internal test (n = 39), and external validation (n = 70). Diagnostic performances of MMD-DL achieved AUCs of 0.85 (95% CI: 0.73, 0.97) and 0.81 (95% CI: 0.73, 0.89) in the test and validation cohorts, respectively, and US radiologists significantly improved their average diagnostic accuracy (57% vs. 60%, P = 0.001) and sensitivity (62% vs. 65%, P < 0.001) by using the AI model for assistance.

Conclusions: The AI model using US videos can provide accurate and reproducible prediction of cervical lymph node metastasis in papillary thyroid carcinoma patients preoperatively, and it can be used as an effective assisting tool to improve diagnostic performance of US radiologists.

Trial registration: We registered on the Chinese Clinical Trial Registry website with the number ChiCTR1900025592.

Keywords: Deep learning; Lymphatic metastasis; Papillary; Thyroid cancer; Ultrasonography.

Fig. 1

Illustration of the multi-scale, multi-frame, and dual-direction deep learning (MMD-DL) model. a Flowchart of the training stages of MMD-DL. b Architecture of the pre-trained feature extractor. c Architecture of MMD-DL with transverse and longitudinal ultrasound videos as inputs and lymph node metastasis probability as the output

Fig. 2

Flowchart of the retrospective and prospective patient enrollment and cohort buildings. a Inclusion and exclusion process of the retrospective patient enrollment. b Inclusion and exclusion process of the multicenter prospective patient enrollment. PTC, papillary thyroid carcinoma; BJTR, Beijing Tong Ren; FMC, the fourth medical center; CJF, China-Japan friendship

Fig. 3

Performances of MMD-DL, radiologists, and radiologists with AI assistance in predicting lymph node metastasis. a Receiver operating characteristic (ROC) curves of MMD-DL in the pre-training and training cohorts. b, c ROC curves of MMD-DL and diagnostic performances of radiologists with and without AI assistance in the test and validation cohorts, respectively. AUC, area under the curve

Fig. 4

Violin plots of the diagnostic accuracy given by a junior, b intermediate, and c senior radiologists with and without AI assistance in the test and validation cohorts together. ACC, accuracy