Construction and validation of machine learning models for predicting lymph node metastasis in cutaneous malignant melanoma: a large population-based study

Abstract

Background: Lymph node status is essential for determining the prognosis of cutaneous malignant melanoma (CMM). This study aimed to develop a machine learning (ML) model for predicting lymph node metastases (LNM) in CMM.

Methods: We gathered data on 6,196 patients from the Surveillance, Epidemiology, and End Results (SEER) database, including known clinicopathologic variables, using six ML algorithms, including logistic regression (LR), support vector machine (SVM), Complement Naive Bayes (CNB), Extreme Gradient Boosting (XGBoost), RandomForest (RF), and k-nearest neighbor algorithm (kNN), to predict the presence of LNM in CMM. Subsequently, we established prediction models. The utilization of the adaptive synthetic (ADASYN) method served to address the challenge posed by imbalanced data. We assessed prediction model performance in terms of average precision (AP), sensitivity, specificity, accuracy, F1 score, precision-recall curves, calibration plots, and decision curve analysis (DCA). Furthermore, employing SHapley Additive exPlanation (SHAP) analysis resulted in the creation of visualized explanations tailored to individual patients.

Results: Among the 6,196 CMM cases, 19.9% (n=1,234) presented with LNM. The XGBoost model showed the best predictive performance when compared with the other algorithms (AP of 0.805). XGBoost showed that age and Breslow thickness were the two most important factors related to LNM.

Conclusions: The XGBoost model predicted LNM of CMM with a high level of precision. We hope that this model could assist surgeons in accurately evaluating surgical approaches and determining the extent of surgery, while also guiding the subsequent adjuvant therapies, thereby improving the prognosis of patients.

Keywords: Cutaneous malignant melanoma (CMM); Surveillance, Epidemiology, and End Results (SEER); lymph node metastasis (LNM); machine learning (ML); shapley additive explanation (SHAP).

Figure 1

Correlation between factors. The depth of color indicates the magnitude of correlation.

Figure 2

Evaluation of the prediction models for lymph node metastases in cutaneous malignant melanoma for the training set (A), and validation set (B). The average precision-recall curves, indicating the trade-off between precision and recall. PR, precision-recall; AP, average precision; XGBoost, Extreme Gradient Boosting; LR, logistic regression; RF, RandomForest; CNB, Complement Naive Bayes; SVM, support vector machine; kNN, the k-nearest neighbor algorithm; CI, confidence interval.

Figure 3

Examples of calibration plots (Brier Score) for predicting lymph node metastases with various models: XGBoost, LR, RF, CNB, SVM, and kNN. The 45° straight line on each graph represents the perfect match between the observed (y-axis) and predicted (x-axis) survival probabilities. A closer distance between two curves indicates greater accuracy. XGBoost, Extreme Gradient Boosting; LR, logistic regression; RF, RandomForest; CNB, Complement Naive Bayes; SVM, support vector machine; kNN, the k-nearest neighbor algorithm; CI, confidence interval.

Figure 4

Decision curves of various models: XGBoost, LR, RF, CNB, SVM, and kNN. XGBoost, Extreme Gradient Boosting; LR, logistic regression; SVM, support vector machine; CNB, Complement Naive Bayes; RF, RandomForest; kNN, the k-nearest neighbor algorithm.

Figure 5

ROC curves of Extreme Gradient Boosting for the training (A), validation (B), and test (C) set. ROC, receiver operating characteristic; AUC, area under the curve; CI, confidence interval.

Figure 6

Summary plots for SHAP values. For each feature, one point corresponds to a single patient. A point’s position along the x-axis represents the impact that feature had on the model’s output for that specific patient. The redder the color indicates that the value is greater, and the bluer the color indicates that the value is smaller. Features are arranged along the y-axis based on their importance, which is given by the mean of their absolute Shapley values. The higher the feature is positioned in the plot, the more important it is for the model. SHAP, SHapley Additive exPlanation.