Bias, precision and statistical power of analysis of covariance in the analysis of randomized trials with baseline imbalance: a simulation study

Abstract

Background: Analysis of variance (ANOVA), change-score analysis (CSA) and analysis of covariance (ANCOVA) respond differently to baseline imbalance in randomized controlled trials. However, no empirical studies appear to have quantified the differential bias and precision of estimates derived from these methods of analysis, and their relative statistical power, in relation to combinations of levels of key trial characteristics. This simulation study therefore examined the relative bias, precision and statistical power of these three analyses using simulated trial data.

Methods: 126 hypothetical trial scenarios were evaluated (126,000 datasets), each with continuous data simulated by using a combination of levels of: treatment effect; pretest-posttest correlation; direction and magnitude of baseline imbalance. The bias, precision and power of each method of analysis were calculated for each scenario.

Results: Compared to the unbiased estimates produced by ANCOVA, both ANOVA and CSA are subject to bias, in relation to pretest-posttest correlation and the direction of baseline imbalance. Additionally, ANOVA and CSA are less precise than ANCOVA, especially when pretest-posttest correlation ≥ 0.3. When groups are balanced at baseline, ANCOVA is at least as powerful as the other analyses. Apparently greater power of ANOVA and CSA at certain imbalances is achieved in respect of a biased treatment effect.

Conclusions: Across a range of correlations between pre- and post-treatment scores and at varying levels and direction of baseline imbalance, ANCOVA remains the optimum statistical method for the analysis of continuous outcomes in RCTs, in terms of bias, precision and statistical power.

Figure 1

Directional bias of statistical methods. Estimates are given at differing levels of baseline-outcome correlation, treatment effect sizes, and baseline imbalance (−1.96, −1.64, −1.28, 0, 1.28, 1.64, 1.96). Estimates derived from ANCOVA represent the unbiased treatment effect. ANOVA – solid line; ANCOVA – dotted line; CSA – dashed line.

Figure 2

Standard errors of statistical methods. Estimates are given at differing levels of baseline-outcome correlation, conditional on treatment effect. The markers show the mean standard error, averaged across the treatment effects. ANOVA – black markers; ANCOVA – grey markers; CSA – white markers.

Figure 3

Power (%) of statistical methods. Estimates are given at differing levels of baseline-outcome correlation, treatment effect sizes, and baseline imbalance (−1.96, −1.64, −1.28, 0, 1.28, 1.64, 1.96). ANOVA – solid line; ANCOVA – dotted line; CSA – dashed line.