Using Machine Learning to Predict Nonachievement of Clinically Significant Outcomes After Rotator Cuff Repair

Abstract

Background: Although some evidence suggests that machine learning algorithms may outperform classical statistical methods in prognosis prediction for several orthopaedic surgeries, to our knowledge, no study has yet used machine learning to predict patient-reported outcome measures after rotator cuff repair.

Purpose: To determine whether machine learning algorithms using preoperative data can predict the nonachievement of the minimal clinically important difference (MCID) of disability at 2 years after rotator cuff surgical repair with a similar performance to that of other machine learning studies in the orthopaedic surgery literature.

Study design: Case-control study; Level of evidence, 3.

Methods: We evaluated 474 patients (n = 500 shoulders) with rotator cuff tears who underwent arthroscopic rotator cuff repair between January 2013 and April 2019. The study outcome was the difference between the preoperative and 24-month postoperative American Shoulder and Elbow Surgeons (ASES) score. A cutoff score was calculated based on the established MCID of 15.2 points to separate success (higher than the cutoff) from failure (lower than the cutoff). Routinely collected imaging, clinical, and demographic data were used to train 8 machine learning algorithms (random forest classifier; light gradient boosting machine [LightGBM]; decision tree classifier; extra trees classifier; logistic regression; extreme gradient boosting [XGBoost]; k-nearest neighbors [KNN] classifier; and CatBoost classifier). We used a random sample of 70% of patients to train the algorithms, and 30% were left for performance assessment, simulating new data. The performance of the models was evaluated with the area under the receiver operating characteristic curve (AUC).

Results: The AUCs for all algorithms ranged from 0.58 to 0.68. The random forest classifier and LightGBM presented the highest AUC values (0.68 [95% CI, 0.48-0.79] and 0.67 [95% CI, 0.43-0.75], respectively) of the 8 machine learning algorithms. Most of the machine learning algorithms outperformed logistic regression (AUC, 0.59 [95% CI, 0.48-0.81]); nonetheless, their performance was lower than that of other machine learning studies in the orthopaedic surgery literature.

Conclusion: Machine learning algorithms demonstrated some ability to predict the nonachievement of the MCID on the ASES 2 years after rotator cuff repair surgery.

Keywords: artificial intelligence; machine learning; rotator cuff repair; rotator cuff tears; shoulder pain.

Figure 1.

SHAP values of the random forest classifier model. This figure provides other relevant information for model interpretation: (1) the predictors are ordered from top to bottom according to their relevance; (2) the more to the right the points of a variable are, the greater the influence of the variable in predicting the outcome (ie, not reaching the minimum MCID); and (3) the redder the point, the higher the predictor value; and the bluer the point, the lower the predictor value. ASES, American Shoulder and Elbow Surgeons; MCID, minimal clinically important difference.