Simulation methods to estimate design power: an overview for applied research

Abstract

Background: Estimating the required sample size and statistical power for a study is an integral part of study design. For standard designs, power equations provide an efficient solution to the problem, but they are unavailable for many complex study designs that arise in practice. For such complex study designs, computer simulation is a useful alternative for estimating study power. Although this approach is well known among statisticians, in our experience many epidemiologists and social scientists are unfamiliar with the technique. This article aims to address this knowledge gap.

Methods: We review an approach to estimate study power for individual- or cluster-randomized designs using computer simulation. This flexible approach arises naturally from the model used to derive conventional power equations, but extends those methods to accommodate arbitrarily complex designs. The method is universally applicable to a broad range of designs and outcomes, and we present the material in a way that is approachable for quantitative, applied researchers. We illustrate the method using two examples (one simple, one complex) based on sanitation and nutritional interventions to improve child growth.

Results: We first show how simulation reproduces conventional power estimates for simple randomized designs over a broad range of sample scenarios to familiarize the reader with the approach. We then demonstrate how to extend the simulation approach to more complex designs. Finally, we discuss extensions to the examples in the article, and provide computer code to efficiently run the example simulations in both R and Stata.

Conclusions: Simulation methods offer a flexible option to estimate statistical power for standard and non-traditional study designs and parameters of interest. The approach we have described is universally applicable for evaluating study designs used in epidemiologic and social science research.

Figure 1

Schematic of the factorial design described in the introduction. Village clusters are first randomized to either control (C) or a sanitation marketing treatment (SAN). Then, households with children < 6 months within each village are randomized to control (C) or to daily nutritional supplementation (NS).

Figure 2

Summary of random effects draws used in a basic simulation of a cluster-level intervention with individual-level outcomes (y_ij). The population mean is μ, b_i are random effects at the cluster level and ε_ij are random errors at the individual level. β₁ is the assumed difference in y_ij for individuals in treated clusters versus individuals in control clusters.

Figure 3

Summary of 2-sided P values obtained from 10,000 simulation runs under a null treatment scenario. The left panel includes a histogram of the P values and the right panel is a quantile-quantile (QQ) plot of the P values against a uniform random variable. The solid line in the QQ plot is the line of equality. Such diagnostic plots - using P values generated in a scenario where the null hypothesis is true - are useful to validate a simulation program.

Figure 4

Power curves (1-Type II error) using the simulation approach versus a conventional analytic formula for a simple cluster-randomized design described in the text.

Figure 5

Power curves (1-Type II error) from the two-treatment, factorial design simulation described in the text. All three treatment effect parameters are assumed to be of equal size.