Accessible model predicts response in hormone receptor positive HER2 negative breast cancer receiving neoadjuvant chemotherapy

Abstract

Hormone receptor-positive/HER2-negative breast cancer (BC) is the most common subtype of BC and typically occurs as an early, operable disease. In patients receiving neoadjuvant chemotherapy (NACT), pathological complete response (pCR) is rare and multiple efforts have been made to predict disease recurrence. We developed a framework to predict pCR using clinicopathological characteristics widely available at diagnosis. The machine learning (ML) models were trained to predict pCR (n = 463), evaluated in an internal validation cohort (n = 109) and validated in an external validation cohort (n = 151). The best model was an Elastic Net, which achieved an area under the curve (AUC) of respectively 0.86 and 0.81. Our results highlight how simpler models using few input variables can be as valuable as more complex ML architectures. Our model is freely available and can be used to enhance the stratification of BC patients receiving NACT, providing a framework for the development of risk-adapted clinical trials.

Fig. 1. Overview of the study and model development workflow.

a Key time points, variables collection and endpoints assessment. b Schematic of the machine learning workflow for model training, evaluation and validation. CONSORT diagram information is embedded in the figure. The model for pCR was trained using 463 patients from three different institutions of a single cancer center and model performance was evaluated on the internal validation dataset (n = 109). The final model was validated to the external validation dataset (n = 151), including patients from three different cancer centers.

Fig. 2. Multidimensional visualization of patients’ characteristics in the different cohorts.

a Radial visualization (RadViz) of patients’ characteristics in the internal cohort. RadViz algorithm plots each point normalizing its value on the axes from the center to the arc. No clear separation according to pCR status was observed. b Parallel coordinates plot for the relation between pCR status and patient’s characteristics in the internal cohort. c UMAP in the internal cohort. Different hyperparameters were used to highlight different possible clusters of patients. Columns correspond to different distances (from left to right: Euclidean, Manhattan and Chebyshev) while rows to different numbers of neighbors (from upper to lower: 10, 20, 50, 100). d Heatmap for patients’ characteristics in the training cohort. A hierarchical clustering algorithm with ward distance was used to cluster features and patients. The higher density of pCR was observed in patients with low ER and high Ki67. e Radial visualization (RadViz) of patients’ characteristics in the external validation cohort. Again, no clear separation according to pCR status was observed. f Parallel coordinates plot for the relation between pCR status and patient’s characteristics in the external validation cohort. g UMAP for the external validation cohort. The hyperparameters used were Manhattan distance and neighbors=10, which provided a good discriminative power in the internal cohort. A cluster of pCR patients was observed, nested in a non-pCR region.

Fig. 3. Model performance, feature importance and external validation of the machine learning model for pCR prediction.

a Comparison of ROC curves in the internal validation set. The best model was Elastic Net trained with SGD solver and calibrated using Platt’s logistic model (sigmoid calibration). The performance of the other models is presented in gray solid lines. The performance of a random classifier is represented in a gray dashed line. The red dot corresponds to the cut-off 0.1925. b Comparison of precision-recall curves in the internal validation set. The Area Under the Precision-Recall Curve (AUC-PR) is a metric used to evaluate the performance of binary classification models, particularly when dealing with imbalanced datasets. It provides a summary measure of the trade-off between the precision and recall for different threshold settings of the classifier. Higher AUC-PR values indicate better overall performance. The baseline performance is represented in a gray dashed line. c Calibration curve. The model showed a fair calibration between 0.1 and 0.4, but systematically tended to be miscalibrated for pCR probabilities greater than 0.4. In the internal validation set, only 4 patients (4%) had a predicted probability above 0.4. d Decision curve analysis and net benefit. The net benefit is a measure that incorporates the true positive rate and false positive rate. Higher net benefit indicates better model performance in terms of clinical decision-making. At low threshold probabilities the model’s net benefit is relatively high, indicating that the model is useful for identifying patients who should be treated. At higher threshold probabilities the net benefit of the model declines. The black dashed line indicates the cut-off 0.1925. e Area under the ROC curve of the Elastic Net model in the external validation set. f Precision-Recall curve of the Elastic Net model in the internal validation set.

Fig. 4. Model performance, feature importance and external validation of the machine learning model for pCR prediction.

a Coefficients of the Elastic Net model. Positive coefficients are plotted clockwise and negative counterclockwise. Age: - 0.036, cN: -0.056, G: 0.059, PR: -0.073, ER: -0.075, cT: -0.075, Ki67: 0.095. b Barplot for SGD Elastic Net permutation importance coefficients. Weights are presented with respective errors. c Barplot of mean SHAP values. The barplot represents the mean global feature importance. The most important features in the dataset were cT and Ki67. d Beeswarn plot for SHAP values. Each patient is represented by a single dot on each feature row. The position of the dot is determined by the SHAP value. Positive SHAP values indicates greater probability of pCR. e SHAP heatmap. In the SHAP heatmap patients are reported on the x-axis and features on the y-axis with SHAP values encoded on a color scale. Patients are ordered in descending order based on the model predicted output and the global importance of each feature is shown on the left barplot. f SHAP decision plot. SHAP decision plots show how the model arrives at its predictions. Each line represents a single patient with corresponding SHAP values for each feature.

Fig. 5. Survival analysis according to pCR and model predictions.

a Kaplan-Meier curves of DFS according to pCR in the whole internal cohort. There were 169 events in pCR No group and 14 events in pCR Yes group. b Kaplan-Meier curves of OS according to pCR in the whole internal cohort. There were 126 events in the pCR No group and 8 events in pCR Yes group. c Kaplan-Meier curves of DFS according to pCR prediction in the internal validation cohort. Predictions were based on the threshold value obtained using MSRS. There were 32 events (57%) in the predicted pCR No group and 12 events (24%) in the predicted pCR Yes group. Since a violation of proportional hazard was graphically suspected and confirmed with Schoenfeld’s test, an exploratory analysis using the p-value for RSMT differences was reported. d Kaplan-Meier curves of OS according to pCR prediction in the internal validation cohort. Predictions were based on the threshold value obtained using MSRS. There were 23 events (40%) in the predicted pCR No group and 9 events (18%) in the predicted pCR Yes group. Again, since a violation of proportional hazard was graphically suspected and confirmed with Schoenfeld’s test, an exploratory analysis using the p-value for RSMT differences was reported. e Kaplan-Meier curves of DFS according to pCR stratified for pCR prediction in patients without pCR in the internal validation cohort. Predictions were based on the threshold value obtained using MSRS. f Kaplan-Meier curves of OS according to pCR stratified for pCR prediction in patients without pCR in the internal validation cohort. Predictions were based on the threshold value obtained using MSRS. MSRS maximally selected rank statistics. RSMT restricted mean survival time.