Using internally developed risk models to assess heterogeneity in treatment effects in clinical trials

Abstract

Background: Recent proposals suggest that risk-stratified analyses of clinical trials be routinely performed to better enable tailoring of treatment decisions to individuals. Trial data can be stratified using externally developed risk models (eg, Framingham risk score), but such models are not always available. We sought to determine whether internally developed risk models, developed directly on trial data, introduce bias compared with external models.

Methods and results: We simulated a large patient population with known risk factors and outcomes. Clinical trials were then simulated by repeatedly drawing from the patient population assuming a specified relative treatment effect in the experimental arm, which either did or did not vary according to a subject's baseline risk. For each simulated trial, 2 internal risk models were developed on either the control population only (internal controls only) or the whole trial population blinded to treatment (internal whole trial). Bias was estimated for the internal models by comparing treatment effect predictions to predictions from the external model. Under all treatment assumptions, internal models introduced only modest bias compared with external models. The magnitude of these biases was slightly smaller for internal whole trial models than for internal controls only models. Internal whole trial models were also slightly less sensitive to bias introduced by overfitting and less sensitive to falsely identifying the existence of variability in treatment effect across the risk spectrum compared with internal controls only models.

Conclusions: Appropriately developed internal models produce relatively unbiased estimates of treatment effect across the spectrum of risk. When estimating treatment effect, internally developed risk models using both treatment arms should, in general, be preferred to models developed on the control population.

Keywords: clinical trial; individualized medicine; risk.

Figure 1

Mean Bias in Treatment Effect. Box plot representing the distribution of mean treatment effect bias (difference in predicted absolute risk reduction (ARR)) from the IWT and ICO models compared to the external model. The box represents the 25th, 50th and 75th percentiles and the whiskers represent the 5th and 95th percentiles

Figure 2

Estimated treatment effects (ARR) across the spectrum of baseline risk. Each sub-figure represents the outcome/risk distribution for the 4 treatment scenarios for each of the three types of models. Estimated treatment effects are on the y-axis and the decile of baseline risk on the X-axis. Error bars signify 95% confidence intervals across the series of simulated trials.

Figure 3

Change in Mean Treatment Effect Bias with change in Overfitting. The x-axis represents the number of events per predictor variable (EPV), and thus the left side of the x-axis represents the most overfit models and the right side the least overfit models. The y-axis represents the mean treatment effect bias (difference in absolute risk reduction) from the IWT and ICO models compared to the external model.