Evaluating the performance of Bayesian and restricted maximum likelihood estimation for stepped wedge cluster randomized trials with a small number of clusters

Abstract

Background: Stepped wedge trials are an appealing and potentially powerful cluster randomized trial design. However, they are frequently implemented with a small number of clusters. Standard analysis methods for these trials such as a linear mixed model with estimation via maximum likelihood or restricted maximum likelihood (REML) rely on asymptotic properties and have been shown to yield inflated type I error when applied to studies with a small number of clusters. Small-sample methods such as the Kenward-Roger approximation in combination with REML can potentially improve estimation of the fixed effects such as the treatment effect. A Bayesian approach may also be promising for such multilevel models but has not yet seen much application in cluster randomized trials.

Methods: We conducted a simulation study comparing the performance of REML with and without a Kenward-Roger approximation to a Bayesian approach using weakly informative prior distributions on the intracluster correlation parameters. We considered a continuous outcome and a range of stepped wedge trial configurations with between 4 and 40 clusters. To assess method performance we calculated bias and mean squared error for the treatment effect and correlation parameters and the coverage of 95% confidence/credible intervals and relative percent error in model-based standard error for the treatment effect.

Results: Both REML with a Kenward-Roger standard error and degrees of freedom correction and the Bayesian method performed similarly well for the estimation of the treatment effect, while intracluster correlation parameter estimates obtained via the Bayesian method were less variable than REML estimates with different relative levels of bias.

Conclusions: The use of REML with a Kenward-Roger approximation may be sufficient for the analysis of stepped wedge cluster randomized trials with a small number of clusters. However, a Bayesian approach with weakly informative prior distributions on the intracluster correlation parameters offers a viable alternative, particularly when there is interest in the probability-based inferences permitted within this paradigm.

Keywords: Bayesian inference; Cluster randomized trial; Intracluster correlation; Restricted maximum likelihood; Simulation study; Stepped wedge.

Fig. 1

Schematic of a stepped wedge design, with N=8 clusters (rows), T=5 periods (columns), and S=2 clusters per treatment sequence (number of times each unique row is repeated)

Fig. 2

Prior distributions for the Bayesian method. Dashed vertical lines indicate the location of the true parameter values chosen for the simulation study (note that the true values for β_j,j=1,…,T depend on the number of periods and so are not shown here)

Fig. 3

Estimated bias for $\widehat{\theta }$ across all trial configurations. See also Table B1, Additional file 1

Fig. 4

Estimated MSE for $\widehat{\theta }$ across all trial configurations. See also Table B2, Additional file 1

Fig. 5

Estimated confidence/credible interval coverage for $\widehat{\theta }$ across all trial configurations. See also Table B3, Additional file 1

Fig. 6

Estimated relative percent error in model-based standard error for $\widehat{\theta }$ across all trial configurations. See also Table B4, Additional file 1

Fig. 7

Widths of estimated confidence/credible intervals for $\widehat{\theta }$ across all trial configurations. See also Table B5, Additional file 1

Fig. 8

Estimated bias for $\widehat{{\rho }_1}$ across all trial configurations. See also Table B6, Additional file 1

Fig. 9

Estimated MSE for $\widehat{{\rho }_1}$ across all trial configurations. See also Table B7, Additional file 1

Fig. 10

Estimated bias for $\widehat{r}$ across all trial configurations. See also Table B8, Additional file 1

Fig. 11

Estimated MSE for $\widehat{r}$ across all trial configurations. See also Table B9, Additional file 1

Fig. 12

Posterior distributions (solid lines) for the treatment effect and within-period intracluster correlation, obtained from the analysis of a randomly-selected simulated dataset for a SW design with S=1,T=5,m=10, true treatment effect θ=0, and intracluster correlation parameters ρ₁=0.1, and r=1. Prior distributions (dashed lines) are overlaid for reference. Note that the prior for the treatment effect is N(0,10⁴) with a probability density of 0.004 at θ=0