Machine learning-based dynamic prediction of lateral lymph node metastasis in patients with papillary thyroid cancer

Abstract

Objective: To develop a web-based machine learning server to predict lateral lymph node metastasis (LLNM) in papillary thyroid cancer (PTC) patients.

Methods: Clinical data for PTC patients who underwent primary thyroidectomy at our hospital between January 2015 and December 2020, with pathologically confirmed presence or absence of any LLNM finding, were retrospectively reviewed. We built all models from a training set (80%) and assessed them in a test set (20%), using algorithms including decision tree, XGBoost, random forest, support vector machine, neural network, and K-nearest neighbor algorithm. Their performance was measured against a previously established nomogram using area under the receiver operating characteristic curve (AUC), decision curve analysis (DCA), precision, recall, accuracy, F1 score, specificity, and sensitivity. Interpretable machine learning was used for identifying potential relationships between variables and LLNM, and a web-based tool was created for use by clinicians.

Results: A total of 1135 (62.53%) out of 1815 PTC patients enrolled in this study experienced LLNM episodes. In predicting LLNM, the best algorithm was random forest. In determining feature importance, the AUC reached 0.80, with an accuracy of 0.74, sensitivity of 0.89, and F1 score of 0.81. In addition, DCA showed that random forest held a higher clinical net benefit. Random forest identified tumor size, lymph node microcalcification, age, lymph node size, and tumor location as the most influentials in predicting LLNM. And the website tool is freely accessible at http://43.138.62.202/.

Conclusion: The results showed that machine learning can be used to enable accurate prediction for LLNM in PTC patients, and that the web tool allowed for LLNM risk assessment at the individual level.

Keywords: central lymph node metastasis; dynamic prediction; feature selection; machine learning; model interpretation; papillary thyroid cancer.

Figure 1

Overview of the analysis workflow.

Figure 2

Flow chart of patient selection.

Figure 3

Feature selection using the LASSO regression model. (A) Lasso regression analysis coefficients. (B) For feature selection, the penalty parameter λ was chosen using the LASSO method, with the minimal mean squared error as the criterion. Dotted vertical lines were drawn on the optimal values and a value of λ of 0.013was chosen, with the optimal λ leading to 30 non-zero coefficients in this study.

Figure 4

Comparisons in model performance between six machine learning and traditional logistic regression-based Nomogram. (A) Receiver operating characteristic curve display a comparison of the predictive model discrimination based on AUC scores. (B) Decision curve analysis assessed the net benefit of the models in terms of clinical utility. The decision curve analysis mapped the net benefit (y-axis) versus the risk threshold (x-axis). It mimicked two scenarios: the black dashed line represented the expected net benefit relative to ‘no intervention’, while the blue dashed line represented the expected net benefit relative to ‘full intervention’. The decision curve analysis indicated that each predictive model had a higher net benefit than the ‘all treatment’ or ‘no treatment’ strategies under different probability thresholds. AUC = area under the curve. (C) The SHAP evaluated a given feature by assessing its contribution to the prediction. The average contribution of the top 20 variables to the magnitude of the model output was ordered according to the descending order of their average absolute contribution to the classification. (D) Each point represents the SHAP value for a particular feature of a particular patient. The further a point is from the x-axis (positive or negative x), the greater the impact of this attribute on the output. The color represents the high (red) and low (blue) original feature values, as indicated by the color array stripes on the right. AST, aspartate aminotransferase; AUC, area under the curve; BMI, body mass index; DBIL, direct bilirubin; T4, tetraiodothyronine; WBC, white blood cell count.

Figure 5

Partial dependent plot (for continuous variables) and box plots (for categorical variables) showing LLNM probabilities vs. variable values for the 30 variables. The y-axis denotes the predicted LLNM probability (range: 0 to 1). The x-axis spans the range (or category) of the 30 predictors. LLNM, lateral lymph node metastases.

Figure 6

Screenshot of examples from the website tool. Input values for key variables to determine the risk of LLNM and show the contribution of each value for the model output. LLNM, lateral lymph node metastases.