Obtaining evidence by a single well-powered trial or several modestly powered trials

Abstract

There is debate whether clinical trials with suboptimal power are justified and whether results from large studies are more reliable than the (combined) results of smaller trials. We quantified the error rates for evaluations based on single conventionally powered trials (80% or 90% power) versus evaluations based on the random-effects meta-analysis of a series of smaller trials. When a treatment was assumed to have no effect but heterogeneity was present, the error rates for a single trial were increased more than 10-fold above the nominal rate, even for low heterogeneity. Conversely, for meta-analyses on a series of trials, the error rates were correct. When selective publication was present, the error rates were always increased, but they still tended to be lower for a series of trials than single trials. We conclude that evidence of efficacy based on a series of (smaller) trials, may lower the error rates compared with using a single well-powered trial. Only when both heterogeneity and selective publication can be excluded, a single trial is able to provide conclusive evidence.

Keywords: Clinical trial; heterogeneity; meta-analysis; publication bias; type I error.