Developing a Bayesian hierarchical model for a prospective individual patient data meta-analysis with continuous monitoring

Abstract

Background: Numerous clinical trials have been initiated to find effective treatments for COVID-19. These trials have often been initiated in regions where the pandemic has already peaked. Consequently, achieving full enrollment in a single trial might require additional COVID-19 surges in the same location over several years. This has inspired us to pool individual patient data (IPD) from ongoing, paused, prematurely-terminated, or completed randomized controlled trials (RCTs) in real-time, to find an effective treatment as quickly as possible in light of the pandemic crisis. However, pooling across trials introduces enormous uncertainties in study design (e.g., the number of RCTs and sample sizes might be unknown in advance). We sought to develop a versatile treatment efficacy assessment model that accounts for these uncertainties while allowing for continuous monitoring throughout the study using Bayesian monitoring techniques.

Methods: We provide a detailed look at the challenges and solutions for model development, describing the process that used extensive simulations to enable us to finalize the analysis plan. This includes establishing prior distribution assumptions, assessing and improving model convergence under different study composition scenarios, and assessing whether we can extend the model to accommodate multi-site RCTs and evaluate heterogeneous treatment effects. In addition, we recognized that we would need to assess our model for goodness-of-fit, so we explored an approach that used posterior predictive checking. Lastly, given the urgency of the research in the context of evolving pandemic, we were committed to frequent monitoring of the data to assess efficacy, and we set Bayesian monitoring rules calibrated for type 1 error rate and power.

Results: The primary outcome is an 11-point ordinal scale. We present the operating characteristics of the proposed cumulative proportional odds model for estimating treatment effectiveness. The model can estimate the treatment's effect under enormous uncertainties in study design. We investigate to what degree the proportional odds assumption has to be violated to render the model inaccurate. We demonstrate the flexibility of a Bayesian monitoring approach by performing frequent interim analyses without increasing the probability of erroneous conclusions.

Conclusion: This paper describes a translatable framework using simulation to support the design of prospective IPD meta-analyses.

Keywords: Bayesian adaptive trial design; Bayesian hierarchical models; Bayesian simulation; COVID-19; International consortium for data sharing; Prospective individual patient data meta-analysis.

Fig. 1

The performance of the five versions of the basic co model: a the distribution of posterior medians of the pooled CCP treatment effect ${\mathrm{\Delta }}_{\mathit{co}}$ and the true value used to generate the data (true value); b boxplots show the median, lower quartile, upper quartile, minimum, and maximum of the number of divergent transitions ($\%$). The proportion of divergent transitions was calculated by (the number of divergent transitions/10,000)$\times 100\%$ in each simulated trial using the five versions of model

Fig. 2

The distribution of posterior medians of pooled CCP treatment effect ${\mathrm{\Delta }}_{\mathit{co}}$ using the final version of basic co model and the extended co model for multi-site RCTs. The black dashed line represents the true value of parameter used to generate the data

Fig. 3

The performance of the extended model for assessing heterogeneity of treatment effect: a prior and 100 posterior distributions of the pooled CCP effect in each level of a pre-specified covariate (${\mathrm{\Delta }}_s$, s = 1,2, or 3), b Distribution of posterior medians of ${\mathrm{\Delta }}_s$ (Based on 4500 simulated trials). The black dashed lines represent the true values of parameters used to generate the data: ${\mathrm{\Delta }}_{s=1}$ = -0.45, ${\mathrm{\Delta }}_{s=2}$ = -0.5, ${\mathrm{\Delta }}_{s=3}$ = -0.6

Fig. 4

The observed cumulative probabilities for the CCP arm and the control treatment arm (CCP arm in observed data: solid purple line with marker $◯$; control arm in observed data: solid orange line with marker $△$) as well as the 95% credible interval (the colored bands) for the predicted cumulative probabilities using the posterior predictive checking: a observed data was generated under proportional cumulative odds assumption, b observed data was generated when the proportional cumulative odds assumption was violated only slightly (Case I), c observed data was generated when the proportional cumulative odds assumption was violated more extremely (Case II)

Fig. 5

At different sample sizes, the proportion of times (out of 2000) in which the stopping rules were reached under the Bayesian monitoring approach. The colored numbers in a can be interpreted as the type 1 error rates at different sample sizes. The colored numbers in b and c can be interpreted as the statistical power