Improved stratification of ALS clinical trials using predicted survival

Abstract

Introduction: In small trials, randomization can fail, leading to differences in patient characteristics across treatment arms, a risk that can be reduced by stratifying using key confounders. In ALS trials, riluzole use (RU) and bulbar onset (BO) have been used for stratification. We hypothesized that randomization could be improved by using a multifactorial prognostic score of predicted survival as a single stratifier.

Methods: We defined a randomization failure as a significant difference between treatment arms on a characteristic. We compared randomization failure rates when stratifying for RU and BO ("traditional stratification") to failure rates when stratifying for predicted survival using a predictive algorithm. We simulated virtual trials using the PRO-ACT database without application of a treatment effect to assess balance between cohorts. We performed 100 randomizations using each stratification method - traditional and algorithmic. We applied these stratification schemes to a randomization simulation with a treatment effect using survival as the endpoint and evaluated sample size and power.

Results: Stratification by predicted survival met with fewer failures than traditional stratification. Stratifying predicted survival into tertiles performed best. Stratification by predicted survival was validated with an external dataset, the placebo arm from the BENEFIT-ALS trial. Importantly, we demonstrated a substantial decrease in sample size required to reach statistical power.

Conclusions: Stratifying randomization based on predicted survival using a machine learning algorithm is more likely to maintain balance between trial arms than traditional stratification methods. The methodology described here can translate to smaller, more efficient clinical trials for numerous neurological diseases.

Keywords: clinical trial stratification; trial methodology predictive analytics.

Figure 1

Experimental approach to testing trial randomization schemes.

Figure 2

Model performance evaluated on a validation dataset. (A) Survival model accuracy was evaluated on the ability of the predictions to accurately stratify into low, medium, and high mortality risk groups, as well as by evaluating the degree of agreement between the predicted survival curves of each of these three groups (red, green, and blue curves) and the observed Kaplan–Meier curves (black stepwise curves). (B) A global evaluation of model performance is performed by plotting the average ROC curve across all folds of the internal tenfold cross‐validation (2B).

Figure 3

Arm balance failures of variables used to define strata per 1000 Simulations. Traditional stratification compared to stratification on tertile log‐likelihood percentiles.

Figure 4

Summary of trial arm balance failure reductions by algorithmic compared to traditional stratification. Two arms of in silico trials were randomized by algorithmic or traditional (riluzole use/bulbar onset) methods as indicated in Figure 1. The indicated quantiles were used for algorithmic randomization.

Figure 5

Balance analysis of baseline features. Traditional stratification compared to stratification on tertile log‐likelihood of survival percentiles for trial sizes ranging from 44 to 448 patients.

Figure 6

Balance analysis of outcomes. Traditional stratification compared to stratification on tertile log‐likelihood of survival percentiles for trial sizes ranging from 44 to 448 patients.

Figure 7

External validation of stratified randomization. A randomization simulation was performed on 279 placebo arm patients from an external, independent clinical trial using both traditional stratified randomization and algorithmic randomization. Predictions were generated using a GBM model trained on PRO‐ACT patient data, and stratified according to the log‐likelihood values corresponding to the tertiles evaluated from PRO‐ACT. A. Arm balance failures of variables used to define strata per 1000 simulations. B. Balance analysis of baseline features per 1000 simulations. C. Balance analysis of outcomes per 1000 simulations.

Figure 8

Plot of power versus sample size for a simulated treatment effect of extension of survival by 2.5 months. A series of simulations to evaluate the ability to detect a treatment effect of a 2.5‐month survival extension across a range of sample sizes. Samples of patients were selected from PRO‐ACT and were stratified and assigned to either a treatment or control arm using either a completely random assignment (green line) stratified by riluzole use and bulbar onset (red line) or by prognostic predicted survival (blue line), and an artificial treatment effect was applied. A simulated treatment effect was applied, and statistics were performed to establish the rate of detection of said treatment effect across a range of sample sizes.