Predicting abatacept retention using machine learning

Abstract

Background: The incorporation of machine learning is becoming more prevalent in the clinical setting. By predicting clinical outcomes, machine learning can provide clinicians with a valuable tool for refining precision medicine approaches and improving treatment outcomes.

Methods: This was a post hoc analysis of pooled patient-level data from the global, real-world ACTION and ASCORE trials in patients with rheumatoid arthritis (RA) initiating abatacept. Patient demographic and disease characteristics were input across 10 machine learning models used to predict 12-month treatment retention. Retention was defined as treatment for > 365 days or ≤ 365 days in patients who achieved remission or major clinical response (based on European Alliance of Associations for Rheumatology response criteria). The pooled dataset was split into a training/validation cohort for model development and a test cohort for an unbiased evaluation of performance. SHapley Additive exPlanation (SHAP) values determined the level of importance and directionality for key patient features predicting abatacept retention.

Results: The pooled ACTION and ASCORE dataset included 5320 patients with RA (mean [standard deviation] age 57.7 [12.7] years; 79% female). The 12-month abatacept retention rate was 61% (n = 3236) with a discontinuation rate of 39% (n = 2037). In the training set (n = 4218), the gradient-boosting classifier model demonstrated the best performance (testing accuracy: 62%). This model had an area under the receiver operating characteristic curve (95% confidence interval) of 0.620 (0.586, 0.653) and F1 score of 0.659 (0.625, 0.689) in the test set of patients (n = 1055). Using this model, the five most important variables predicting 12-month abatacept retention were low body mass index (BMI), low American College of Rheumatology functional status class, anti-citrullinated protein antibody (ACPA) positivity, low Patient Global Assessment, and younger age.

Conclusions: The gradient-boosting classifier model identified key patient features predictive of abatacept retention from this large, real-world study population. The SHAP values conveyed the directionality and importance of BMI, functional status, ACPA serostatus, Patient Global Assessment, and age for abatacept retention. Findings are consistent with previous observations and help validate the machine learning approach for predictive modelling in RA treatment, and may help inform clinical decision making.

Trial registration: NCT02109666 (ACTION), NCT02090556 (ASCORE).

Keywords: Abatacept; Machine learning; Retention; Rheumatoid arthritis; Treatment response.

Fig. 1

Gradient-boosting classifier variable importance for predicting abatacept retention at (a) 12 and (b) 6 months. The numerical values of variable importance in a gradient-boosting classifier provide insights into the relevance of each variable within the model's decision-making process; for example, the higher the numerical values of variable importance, the more relevant the variable (ie, corticosteroid dose) to the outcome (ie, predicting abatacept retention). *Evaluable for secondary analysis of clinical efficacy. ACPA anti-citrullinated protein antibody, ACR American College of Rheumatology, BMI body mass index, CDAI Clinical Disease Activity Index, CRP C-reactive protein, DAS28 Disease Activity Score in 28 joints, ESR erythrocyte sedimentation rate, HAQ-DI Health Assessment Questionnaire-disability index, MTX methotrexate, RA rheumatoid arthritis, RF rheumatoid factor, SDAI Simplified Disease Activity Index, SJC28 swollen joint count in 28 joints, TJC28 tender joint count in 28 joints, TNFi tumor necrosis factor inhibitor

Fig. 2

Overall SHAP value plot after (a) 12 months and (b) 6 months. Figure shows the most important variables in the ACTION/ASCORE combined dataset that were predictive of abatacept retention. Colors indicate the value of the variable: red represents higher numerical values of the variable and blue represents lower numerical values. The bulges in the plot indicate more patients with that value; each dot represents a single patient. Higher SHAP values indicate a higher likelihood of retention. *Evaluable for secondary analysis of clinical efficacy. ACPA anti-citrullinated protein antibody, ACR American College of Rheumatology, BMI body mass index, CDAI Clinical Disease Activity Index, CRP C-reactive protein, DAS28, Disease Activity Score in 28 joints, ESR erythrocyte sedimentation rate, HAQ-DI Health Assessment Questionnaire-disability index, MTX methotrexate, RA rheumatoid arthritis, RF rheumatoid factor, SDAI Simplified Disease Activity Index, SHAP SHapley Additive exPlanation, SJC28 swollen joint count in 28 joints, TJC28 tender joint count in 28 joints, TNFi tumor necrosis factor inhibitor

Fig. 3

Individual SHAP value plots for top 5 characteristics predictive of abatacept 12-month retention. Colors indicate the value of the variable: red represents higher and blue represents lower. Each dot represents a single patient. Higher SHAP values indicate a higher likelihood of retention. *For these characteristics, more columns are included here compared to the overall SHAP plot (Fig. 2) due to the step of filling missing values. Missing values are filled as predicted from available values providing a numerical output (logit); the numerical output (logit) is used to make the prediction to prevent the information loss caused by the step of transferring into binary outputs. ACPA anti-citrullinated protein antibody, ACR American College of Rheumatology, BMI body mass index, SHAP SHapley Additive exPlanations

Fig. 4

Individual SHAP value plots for top five characteristics predictive of abatacept 6-month retention. Colors indicate the value of the variable: red represents higher and blue represents lower. Each dot represents a single patient. Higher SHAP values indicate a higher likelihood of retention. *For these characteristics, more columns are included here compared to the overall SHAP plot (Fig. 2) due to the step of filling missing values. Missing values are filled as predicted from available values providing a numerical output (logit); the numerical output (logit) is used to make the prediction to prevent the information loss caused by the step of transferring into binary outputs. †Breaks or steps in the SHAP plots likely result from use of the gradient boosting classifier, a decision tree-based model, which uses age at certain values to split the tree decisions in the model. ACPA anti-citrullinated protein antibody, ACR American College of Rheumatology, MTX methotrexate, SHAP SHapley Additive exPlanations