Development and Evaluation of a Simulation-Based Algorithm to Optimize the Planning of Interim Analyses for Clinical Trials in ALS

Abstract

Background and objectives: Late-phase clinical trials for neurodegenerative diseases have a low probability of success. In this study, we introduce an algorithm that optimizes the planning of interim analyses for clinical trials in amyotrophic lateral sclerosis (ALS) to better use the time and resources available and minimize the exposure of patients to ineffective or harmful drugs.

Methods: A simulation-based algorithm was developed to determine the optimal interim analysis scheme by integrating prior knowledge about the success rate of ALS clinical trials with drug-specific information obtained in early-phase studies. Interim analysis schemes were optimized by varying the number and timing of interim analyses, together with their decision rules about when to stop a trial. The algorithm was applied retrospectively to 3 clinical trials that investigated the efficacy of diaphragm pacing or ceftriaxone on survival in patients with ALS. Outcomes were additionally compared with conventional interim designs.

Results: We evaluated 183-1,351 unique interim analysis schemes for each trial. Application of the optimal designs correctly established lack of efficacy, would have concluded all studies 1.2-19.4 months earlier (reduction of 4.6%-57.7% in trial duration), and could have reduced the number of randomized patients by 1.7%-58.1%. By means of simulation, we illustrate the efficiency for other treatment scenarios. The optimized interim analysis schemes outperformed conventional interim designs in most scenarios.

Discussion: Our algorithm uses prior knowledge to determine the uncertainty of the expected treatment effect in ALS clinical trials and optimizes the planning of interim analyses. Improving futility monitoring in ALS could minimize the exposure of patients to ineffective or harmful treatments and result in significant ethical and efficiency gains.

Figure 1. Schematic Illustration of the Simulation-Based Algorithm to Optimize Interim Analysis Schemes

First, the investigators decide on the range of acceptable interim analysis schemes. Previous research is then used to formulate the uncertainty around the treatment effect and the probability of trial success. In the next step, each interim analysis scheme is evaluated by means of simulation based on the trial assumptions made. Finally, the best performing interim analysis scheme is identified, based on a selection criterion (e.g., shortest average trial duration, smallest average sample size, or shortest average drug exposure). A script to replicate the simulation is provided in eAppendix 1 (links.lww.com/WNL/C768).

Figure 2. Retrospective Application of the Algorithm in the 3 ALS Clinical Trials

Performance of the optimized interim analysis schemes when applied retrospectively to the DiPALS (A), RespiStimALS (B), and Ceftriaxone-ALS (C) trials. The black dashed line reflects the development of the test statistic over time, which is calculated sequentially and represents the treatment evidence based on the data accumulated thus far. As soon as the treatment evidence surpasses the red line, the trial can be stopped for futility. This line, therefore, represents the decision rules of the trial. In addition, a horizontal dotted line was plotted to indicate whether treatment evidence favors treatment or control. The blue triangle represents the stopping decision of the optimized interim analysis scheme, whereas the red diamond reflects when the trial was actually stopped. ALS = amyotrophic lateral sclerosis; DMEC = Data Monitoring and Ethics Committee.

Figure 3. Development of the Primary Outcome in the DiPALS Trial

Development of the observed treatment effect over time at each interim analysis (A–D). For illustrative purposes, we show the development of the Kaplan-Meier curves at each interim analysis of the DiPALS trial. At the first interim analysis (A), the hazard ratio (HR) is 2.15 (0.75–6.22), and the trial can be stopped (Figure 2A). Taking into account the accumulated data at this time, and the expected HR of 0.45, the probability that the final analysis may still yield statistical significance in favor of treatment is only 6.3%.

Figure 4. Behavior of the Optimized Interim Analysis Scheme Under Different Treatment Effects for Ceftriaxone

We simulated the effect of ceftriaxone from very beneficial to very harmful (10,000 simulations per scenario), while keeping the randomization ratio, enrollment rate, and survival in the placebo arm fixed. For each scenario, we determined when the trial could be stopped, using the optimized interim analysis scheme (green), a conventional interim design (red), and a design without interim analyses (gray). An interim analysis scheme is most effective when there is a large positive or negative treatment effect when compared with a design without interim analyses (A). In addition, we illustrated the probability that a trial conclusion would result in a more than 20% reduction in trial duration, using the optimized (green) and conventional (red) interim designs vs a design without interim analyses (B). Conventional interim design incorporated a single interim analysis at 60% of the maximum number of events using O'Brien-Fleming–type decision rules.