Using machine learning algorithms to optimize treatment with high-cost biologics in a national cohort of patients with inflammatory bowel disease

Abstract

Objectives: Prediction models using statistical or machine learning (ML) approaches can enhance clinical decision support tools. Infliximab (IFX), a biologic with a newly introduced biosimilar for Crohn's disease (CD) and ulcerative colitis (UC), presents an opportunity to evaluate these tools at time of biosimilar switch to predict disease flares. This study sought to evaluate real-world safety and effectiveness of nonmedical IFX biosimilar switch in a national US cohort of CD and UC patients, and to develop and compare interpretable models for predicting adverse clinical events among patients on maintenance IFX.

Materials and methods: This retrospective cohort study used administrative and clinical data from the National Veterans Health Administration Corporate Data Warehouse. It included 2529 Veterans with CD or UC on maintenance IFX (2017-2020), either continuing originator IFX or switching to a biosimilar. The primary outcome was disease-related flare. Classification and survival models were developed using traditional and ML methods and assessed via receiver operating characteristic curve, precision-recall curve, and decision curve analysis.

Results: In 2529 Veterans with CD or UC, biosimilar switch had low predictive importance across survival models. Objective laboratory-related information yielded the highest validation. Random forest+ (RF+) outperformed all other statistical and ML models. Prior flares and total health-care encounters were the 2 most important predictors, while hemoglobin was the top laboratory predictor.

Conclusions: Prediction models, particularly RF+, may aid in optimizing biologic therapy for CD and UC by identifying patients at higher risk of flare following a biosimilar switch.

Keywords: artificial intelligence; biologic products; biosimilar pharmaceuticals; inflammatory bowel diseases; machine learning.

Figure 1.

Validation AUROC prediction performance for classification models. (A) We summarize the validation AUROC prediction performance for predicting IBD-related flares using various prediction models (column) with different sets of predictor variables (color). Results are shown for different forecasting (x-axis) and accrual periods (row). Higher values indicate more accurate prediction performance. Lines represent the mean validation AUROC, averaged across 50 random training–validation–test splits. Standard errors are shown as shaded regions around the mean estimates. Across all forecasting and accrual periods, incorporating laboratory-related information through the mean, max, and most recent lab values (orange) generally yields the highest validation AUROC. (B) Moreover, we show the distribution of pairwise differences between the validation AUROC from RF+ including the mean, max, and most recent lab data and that from competitor methods (x-axis) across different forecasting (columns) and accrual periods (rows). Across the various forecasting and accrual periods, RF+ yields significantly higher validation prediction accuracy than the other competitor methods. Asterisks represent the strength of statistical significance from a permutation test that formally assesses whether RF+ yields higher prediction accuracy than the competitor method (ie, whether the differences are significantly greater than 0). Specifically, ***P< .001, **P<.01, and *P <.05. Results are shown using 50 random training–validation–test splits. Abbreviations: AUROC, area under the receiver operating characteristic curve; IBD, inflammatory bowel disease; RF, random forest.

Figure 2.

RF+ calibration and decision curve analysis. (A) The number of observed flares closely matches the number of predicted flares from our best classification model, RF+ including the mean, max, and most recent lab data, across different forecasting (color) and accrual (shape) periods. The dashed line represents perfect calibration. (B) Using a decision curve analysis, we evaluated the net benefit of the RF+ model compared to that from the logistic regression model, treating all patients, or treating no patients. The net benefit (y-axis) is shown across different threshold probabilities (x-axis) for different forecasting (columns) and accrual periods (rows). The dotted line represents the threshold probability corresponding to the optimal sensitivity and specificity for the RF+ model. RF+ yields higher net benefit than the other strategies across a range of reasonable threshold probabilities. Abbreviation: RF, random forest.

Figure 3.

Most important features from RF+ for predicting IBD-related flares. (A) For each forecasting (columns) and accrual (rows) period, we measured the mean decrease in impurity+ (MDI+) importance (black) of each feature in the RF+ including the mean, max, and most recent lab data. Features are ordered by their mean total MDI+ importance with higher values indicating greater importance. Additionally, for each feature, we quantified the importance of their linear (red) and nonlinear (blue) contributions in the RF+ model. Each point represents the mean importance of a feature across 50 random training–validation–test splits, and error bars represent the standard errors. The top 15 most important features are shown for each choice of forecasting and accrual period. (B) Partial dependence plots are shown for the 6 most important features from the RF+ model including the mean, max, and most recent labs for the 1-year accrual, 3-month forecasting model. For each of these important features, the partial dependence plot shows the average predicted probability (or risk) of flaring across different values of that feature while keeping all other features fixed. This reveals whether the predicted risk of flaring increases or decreases as values of the feature (eg, # of prior flares) increase. The partial dependence functions are summarized across 50 random training–validation–test splits with the black solid line representing the mean and the shaded region representing the inner 95% quantiles of this distribution. Abbreviations: IBD, inflammatory bowel disease; RF, random forest.