Design and Evaluation of an External Control Arm Using Prior Clinical Trials and Real-World Data

Abstract

Purpose: We discuss designs and interpretable metrics of bias and statistical efficiency of "externally controlled" trials (ECT) and compare ECT performance to randomized and single-arm designs.

Experimental design: We specify an ECT design that leverages information from real-world data (RWD) and prior clinical trials to reduce bias associated with interstudy variations of the enrolled populations. We then used a collection of clinical studies in glioblastoma (GBM) and RWD from patients treated with the current standard of care to evaluate ECTs. Validation is based on a "leave one out" scheme, with iterative selection of a single-arm from one of the studies, for which we estimate treatment effects using the remaining studies as external control. This produces interpretable and robust estimates on ECT bias and type I errors.

Results: We developed a model-free approach to evaluate ECTs based on collections of clinical trials and RWD. For GBM, we verified that inflated false positive error rates of standard single-arm trials can be considerably reduced (up to 30%) by using external control data.

Conclusions: The use of ECT designs in GBM, with adjustments for the clinical profiles of the enrolled patients, should be preferred to single-arm studies with fixed efficacy thresholds extracted from published results on the current standard of care.

Figure 1:

Bias (π_SAT − π_HC)and deviations from a targeted type I error rate of 10%. Bias is due to different patient populations in the single arm trial (SAT) and in the historical study. A single binary characteristic (X=1 or X=0) correlates with the binary outcome Y, and the experimental treatment has no therapeutic effect Pr(Y|X, A = 1) = Pr(Y|X, A = 0). The characteristic X=1 was present in 50% of the patients in the historical control arm, P_HC(X = 1) = 0.5. Panel (A) shows the difference (π_SAT − π_HC) for a range of probabilities P_SAT(X = 1). We consider four levels of association between X and Y; (Pr(Y|X = 1, A = a) and P(Y|X = 0, A = a) equal either to (0.3, 0.9), (0.4, 0.8), (0.5, 0.7) or (0.6, 0.6). Panel (B) indicates, for a SAT (with standard z-test for proportions, H₀: π_SAT ≤ π_HC) how the false positive rate (y-axis) of the design deviates without adjustments from the targeted type I error rate of 10% when the prevalence P_SAT(X = 1) = 0.3, 0.5, 0.6 or 0.8. We consider different sample sizes (x-axis) of the SAT. In panel (B) we assume to know the parameter π_HC.

Figure 2:

Treatment effect estimates of the ECT design. For each studies the RT+TZM arm was used as ECT’s experimental arm and (after adjustment for patents characteristics) compared to the RT+TZM arms of the remaining four studies. Panel (A) shows, for each of the study, covariate adjusted treatment effect estimates (point estimates and 90% confidence interval, n equals to the arm-specific size). Panel (B) shows treatment effect estimates (average value, 5th and 95th percentile) across 10,000 subsamples of n=46 patients using different adjustment methods. We consider direct standardization, matching, inverse-probability weighting and marginal structural models (DSM, PS-M, IPW, MSM). For IPW and MSM we use different reference distributions Pr_X(x) (see expression 1) of pre-treatment characteristics X. Panel (C) shows the distribution of treatment effect estimates of the ECT (blue line) and RCT (black line) across subsamples of n=46 patients.

Figure 3:

Model-based evaluation of the type I error and power for RCT, ECT and single-arm trial (SAT) designs for an overall study sample size of n=20, …, 160 patients. In the model-based approach (Supplementary-Material) we sampled baseline characteristics X from the five studies in Table 1, and generated outcomes Y from models Pr(Y|X, A). Panel (A) shows for all studies the type I error rates of RCT, ECT and SAT designs at different overall sample sizes. Different line types (solid, dashed, dotted, etc.) indicate different studies (Table 1). Panels (B-F) show for each study, the power of RCT, SAT and ECT designs, and sample size to achieve 80% power (dotted vertical lines). In panel A, the single arm trial experimental outcomes have been generated as in the ECT simulations, but outcomes Y are directly compared to the EORTC-NCIC CE.3 study estimates, without adjustments for different distributions of patients’ characteristics. For RCTs, half of the randomly selected profiles X are used to define the experimental arm and the remaining half defines the control arm. Two-group (RCT) and single-group (single arm trial) z-tests for proportions were used for testing. To compute the power in Panels B-F of the SAT, we assumed that the historical control benchmark π_HC was correctly specified.