Statistical Properties of Stepped Wedge Cluster-Randomized Trials in Infectious Disease Outbreaks

Abstract

Randomized controlled trials are crucial for the evaluation of interventions such as vaccinations, but the design and analysis of these studies during infectious disease outbreaks is complicated by statistical, ethical, and logistical factors. Attempts to resolve these complexities have led to the proposal of a variety of trial designs, including individual randomization and several types of cluster randomization designs: parallel-arm, ring vaccination, and stepped wedge designs. Because of the strong time trends present in infectious disease incidence, however, methods generally used to analyze stepped wedge trials might not perform well in these settings. Using simulated outbreaks, we evaluated various designs and analysis methods, including recently proposed methods for analyzing stepped wedge trials, to determine the statistical properties of these methods. While new methods for analyzing stepped wedge trials can provide some improvement over previous methods, we find that they still lag behind parallel-arm cluster-randomized trials and individually randomized trials in achieving adequate power to detect intervention effects. We also find that these methods are highly sensitive to the weighting of effect estimates across time periods. Despite the value of new methods, stepped wedge trials still have statistical disadvantages compared with other trial designs in epidemic settings.

Keywords: cluster-randomized trials; epidemics; permutation tests; simulation; stepped wedge trials; synthetic control; vaccine trials.

Figure 1

Estimates by model for an analysis of vaccine efficacy (VE) in an outbreak setting. Median and interquartile range (IQR) of VE estimates for direct VE of 0.6 (vertical line) and a basic reproduction number, R₀, of 2.47 for individually randomized trial (IRT) analyzed with stratified Cox model, cluster-randomized trial (CRT) analyzed with a Cox model with a gamma-distributed shared frailty, and stepped wedge trials with 4 clusters crossing over every 28 days (SWT-A) (A) and with 1 cluster crossing over every 7 days (SWT-B) (B) analyzed by a Cox model with a gamma-distributed shared frailty (SWT-PH), mixed effects model (MEM), mixed effects model with cluster-period random effect (MEM-CP), nonparametric within-period method equally weighted across periods (NPWP-1) and weighted across periods by total case count (NPWP-2), and synthetic control method equally weighted across clusters and periods (SC-1), equally weighted across clusters and weighted across periods by total case count (SC-2), weighted across clusters by inverse mean square prediction error and equally weighted across periods (SC-Wt-1), and weighted across clusters by inverse mean square prediction error and across periods by total case count (SC-Wt-2).

Figure 2

Power and type I error by model for an analysis of vaccine efficacy in an outbreak setting. Empirical power for direct vaccine effect of 0.6 (A) and empirical type I error for direct vaccine effect of 0 (B) with a basic reproduction number, R₀, of 2.47 for individually randomized trial analyzed with stratified Cox model (IRT), cluster-randomized trial analyzed with a Cox model with a gamma-distributed shared frailty (CRT), and stepped wedge trials with 4 clusters crossing over every 28 days (SWT-A) and with 1 cluster crossing over every 7 days (SWT-B) analyzed by a Cox model with a gamma-distributed shared frailty (SWT-PH), mixed effects model (MEM), mixed effects model with cluster-period random effect (MEM-CP), nonparametric within-period method equally weighted across periods (NPWP-1) and weighted across periods by total case count (NPWP-2), and synthetic control method equally weighted across clusters and periods (SC-1), equally weighted across clusters and weighted across periods by total case count (SC-2), weighted across clusters by inverse mean square prediction error and equally weighted across periods (SC-Wt-1), and weighted across clusters by inverse mean square prediction error and across periods by total case count (SC-Wt-2). Four type I error values greater than 10% are denoted by “+” in (B): SWT-A MEM (72%), SWT-B MEM (72%), SWT-A MEM-CP (28%), and SWT-B MEM-CP (52%). The horizontal line in (B) denotes the nominal type I error of 5%.

Figure 3

Estimates and power by basic reproduction number, R₀, and model for an analysis of vaccine efficacy (VE) in an outbreak setting for direct VE of 0.6. Median VE estimate (A) and empirical power (B) for direct VE of 0.6 (horizontal line in (A)) versus R₀, for individually randomized trial (IRT) analyzed with stratified Cox model (squares, solid line), cluster-randomized trial (CRT) analyzed with a Cox model with a gamma-distributed shared frailty (circles, dashed line), and stepped wedge trials (SWT-PH) with 4 clusters crossing over every 28 days (SWT-A, filled points) and with 1 cluster crossing over every 7 days (SWT-B, unfilled points) analyzed by a Cox model with a gamma-distributed shared frailty (diamonds, dotted line), nonparametric within-period method weighted across periods by total case count (NPWP-2; upward triangle, dash-dotted line), and synthetic control method equally weighted across clusters and weighted across periods by total case count (SC-2; downward triangle, long-dash line).

Figure 4

Estimates and power by basic reproduction number, R₀, and model for an analysis of vaccine efficacy (VE) in an outbreak setting for direct VE of 0.8. Median VE estimate (A) and empirical power (B) for direct VE of 0.8 (horizontal line in (A)) versus R₀, for individually randomized trial (IRT) analyzed with stratified Cox model (squares, solid line), cluster-randomized trial (CRT) analyzed with a Cox model with a gamma-distributed shared frailty (circles, dashed line), and stepped wedge trials with 4 clusters crossing over every 28 days (SWT-A, filled points) and with 1 cluster crossing over every 7 days (SWT-B, unfilled points) analyzed by a Cox model with a gamma-distributed shared frailty (SWT-PH; diamonds, dotted line), nonparametric within-period method weighted across periods by total case count (NPWP-2; upward triangle, dash-dotted line), and synthetic control method equally weighted across clusters and weighted across periods by total case count (SC-2; downward triangle, long-dash line).

Figure 5

Estimates and type I error by basic reproduction number, R₀, and model for an analysis of vaccine efficacy (VE) in an outbreak setting for direct VE of 0. Median VE estimate (A) and empirical type I error (B) for direct VE of 0 (horizontal line in (A)) versus R₀, for individually randomized trial (IRT) analyzed with stratified Cox model (squares, solid line), cluster-randomized trial (CRT) analyzed with a Cox model with a gamma-distributed shared frailty (circles, dashed line), and stepped wedge trials with 4 clusters crossing over every 28 days (SWT-A, filled points) and with 1 cluster crossing over every 7 days (SWT-B, unfilled points) analyzed by a Cox model with a gamma-distributed shared frailty (SWT-PH; diamonds, dotted line), nonparametric within-period method weighted across periods by total case count (NPWP-2; upward triangle, dash-dotted line), and synthetic control method equally weighted across clusters and weighted across periods by total case count (SC-2; downward triangle, long-dash line). The horizontal line in (B) denotes the nominal type I error of 5%.

Figure 6

Period-specific estimates by period and model for an analysis of vaccine efficacy (VE) in an outbreak setting. Average period-specific VE estimate by period for stepped wedge trials with 4 clusters crossing over every 28 days (SWT-A) (A) and 1 cluster crossing over every 7 days (SWT-B) (B) with VE = 0.6 and a basic reproduction number, R₀, of 2.47, analyzed by nonparametric within-period method (NPWP), synthetic control (SC) method weighted equally across clusters, and SC method weighted across clusters by inverse mean square prediction error (SC-Wt). Values less than −0.2 are truncated to −0.2 for legibility.