Using Clinical Trial Simulators to Analyse the Sources of Variance in Clinical Trials of Novel Therapies for Acute Viral Infections

Abstract

Background: About 90% of drugs fail in clinical development. The question is whether trials fail because of insufficient efficacy of the new treatment, or rather because of poor trial design that is unable to detect the true efficacy. The variance of the measured endpoints is a major, largely underestimated source of uncertainty in clinical trial design, particularly in acute viral infections. We use a clinical trial simulator to demonstrate how a thorough consideration of the variability inherent in clinical trials of novel therapies for acute viral infections can improve trial design.

Methods and findings: We developed a clinical trial simulator to analyse the impact of three different types of variation on the outcome of a challenge study of influenza treatments for infected patients, including individual patient variability in the response to the drug, the variance of the measurement procedure, and the variance of the lower limit of quantification of endpoint measurements. In addition, we investigated the impact of protocol variation on clinical trial outcome. We found that the greatest source of variance was inter-individual variability in the natural course of infection. Running a larger phase II study can save up to $38 million, if an unlikely to succeed phase III trial is avoided. In addition, low-sensitivity viral load assays can lead to falsely negative trial outcomes.

Conclusions: Due to high inter-individual variability in natural infection, the most important variable in clinical trial design for challenge studies of potential novel influenza treatments is the number of participants. 100 participants are preferable over 50. Using more sensitive viral load assays increases the probability of a positive trial outcome, but may in some circumstances lead to false positive outcomes. Clinical trial simulations are powerful tools to identify the most important sources of variance in clinical trials and thereby help improve trial design.

Fig 1. Viral load curves over time of 50 (a) and 100 (b) patients.

If there are less curves than simulated patients, the infection did not take off in the remaining number of patients (R₀ < 1) (curve is a flat line across the x-axis). x-axis: time in days, y-axis: viral load in particles per ml.

Fig 2. Variance of viral load measurements.

For this figure we simulated influenza virus infection in 1000 patients that all had the same natural course of infection. For each of the patients, viral load measurements were generated, simulating qPCR (upper row) and TCID₅₀ assays (lower row). The variance of the TCID₅₀ assay (lognormal distribution) is greater than that of the qPCR assay (Poisson distribution). The variance of the viral load assays is small compared to the variation in natural infection among patients. Upper row: qPCR measurements, unit: viral cDNA/ml. Lower row: TCID₅₀ measurements, unit TCID₅₀/ml. a, c: True (simulated) viral load curves of 1000 patients with same course of natural infection. b, d: simulated measurements of 1000 patients with the same natural course of infection. b: lower limit of quantification of the PCR assay was 3.33 log₁₀ cDNA/ml (2137 cDNA/ml), d: lower limit of quantification of TCID₅₀ assay was 2 log₁₀ TCID₅₀/ml (100 TCID₅₀/ml).

Fig 3. Experiment 1: Individual variability in natural infection.

The plots show the number of successful trials out of 100 simulated trial runs (y-axis) for different assumed efficacies of the treatment (x-axis). All 100 iterations for each assumed efficacy value had exactly the same setup and differed only in the random number seed. The number of successful trials out of 100 runs can be interpreted as the power of the trial. The power of the trial depends on the mechanism of action of the treatment and the number of patients in the trial. As individual variability in natural infection is large, trials with 50 patients do not reach a power of 80%, even if the assumed efficacy of the treatment is high (90+%). Coloured lines show the number of successful trials out of 100 runs depending on efficacy for different endpoint measurements (PCR: viral load AUC measured with qPCR; Symptom: temperature AUC; TCID: viral load AUC measured with TCID₅₀; Simulated: viral load AUC of the simulated viral load AUC). Upper row: trials with 50 patients. Lower row: trials with 100 patients. a, e: treatment acts on all stages of the virus life cycle/model parameters. b, f: treatment acts on the infection rate. c, g: treatment acts on the virus production rate. d, h: treatment acts on the virus clearance rate.

Fig 4. Experiment 5: Sensitivity of viral load assay (qPCR).

Plots show the number of successful trials out of 100 runs (y-axis) over the assumed mean efficacy of treatment (x-axis). The probability of success corresponds to the power of the trial. The parameters determining the course of natural infection were drawn from the same random number distributions for each patient as explained in the main text. The efficacy for each patient (response) was fixed to the same value for each patient in each run. Thin coloured lines show the power of the trial dependent on the efficacy of the treatment for qPCR viral load assays assuming different lower limits of quantification. The bold red line shows the power of the trial dependent on the efficacy of the treatment, if the simulated viral load curve is considered. Very insensitive assays can greatly reduce the power of a trial, especially in potent drugs that act on several stages of the virus life cycle (a). Conversely, if the treatment acts on the infection rate (b) or the virus production rate (c), very sensitive viral load assays tend to give false positive results. Trials with 100 patients. a: treatment acts on all model parameters. b: treatment acts on the infection rate. c: treatment acts on the virus production rate. d: treatment acts on the virus clearance rate.

Fig 5. Experiment 6: Day of Treatment. Treatment acts on all model parameters.

Plots show the number of successful trials out of 100 runs (y-axis) over the assumed mean efficacy of treatment (x-axis). The probability of success corresponds to the power of the trial. The parameters determining the course of natural infection were drawn from the same random number distributions for each patient as explained in the main text. The efficacy for each patient (response) was fixed to the same value for each patient in each run. The later treatment is given, the lower the power of the trial. a: treatment on day 1; b: treatment on day 2; c: treatment on day 3. Coloured lines show power of the trial depending on efficacy for different endpoint measurements (PCR: viral load AUC measured with qPCR; Symptom: temperature AUC; TCID: viral load AUC measured with TCID₅₀; True: viral load AUC of the simulated viral load AUC).