A prediction model for asthma exacerbations after stopping asthma biologics

Abstract

Background: Little is known regarding the prediction of the risks of asthma exacerbation after stopping asthma biologics.

Objective: To develop and validate a predictive model for the risk of asthma exacerbations after stopping asthma biologics using machine learning models.

Methods: We identified 3057 people with asthma who stopped asthma biologics in the OptumLabs Database Warehouse and considered a wide range of demographic and clinical risk factors to predict subsequent outcomes. The primary outcome used to assess success after stopping was having no exacerbations in the 6 months after stopping the biologic. Elastic-net logistic regression (GLMnet), random forest, and gradient boosting machine models were used with 10-fold cross-validation within a development (80%) cohort and validation cohort (20%).

Results: The mean age of the total cohort was 47.1 (SD, 17.1) years, 1859 (60.8%) were women, 2261 (74.0%) were White, and 1475 (48.3%) were in the Southern region of the United States. The elastic-net logistic regression model yielded an area under the curve (AUC) of 0.75 (95% confidence interval [CI], 0.71-0.78) in the development and an AUC of 0.72 in the validation cohort. The random forest model yielded an AUC of 0.75 (95% CI, 0.68-0.79) in the development cohort and an AUC of 0.72 in the validation cohort. The gradient boosting machine model yielded an AUC of 0.76 (95% CI, 0.72-0.80) in the development cohort and an AUC of 0.74 in the validation cohort.

Conclusion: Outcomes after stopping asthma biologics can be predicted with moderate accuracy using machine learning methods.

Figure 1.. Workflow of the training, validation procedure and selection of final machine learning model.

The development cohort was randomly divided into 10 equal and independent parts (folds): models were trained on 9 folds and tested on the holdout fold. The best model tuning parameters was selected through grid search and the model evaluated on the holdout fold. This procedure was repeated 10 times until each fold has been used for training and testing. The performance results over the cross-validations were then averaged for reporting. The final model was obtained by training the model on the 10-folds combined using the overall best tuning parameters from the cross-validation and evaluated on the 20% holdout validation cohort. The best parameters were selected based on the highest AUC performance.

Figure 2A.

ROC Curve in Development Cohort

Figure 2B.

ROC Curve in Validation Cohort

Figure 3.. SHAP summary plot for the top 20 variables of the RF model contributing to predict risk of no asthma exacerbation.

Positive SHAP values (x-axis) indicate the variable contribute to push the model to make no asthma exacerbation, while negative values indicate the variable contribute to push the model to make asthma exacerbation predictions. Variables on the y-axis are sorted by the sum of the absolute SHAP values over all patients. The values adjacent to each variable are the corresponding mean absolute SHAP value for the variable. Each plotted point represents a patient, and the color represents the variable value: red indicates high and green low.