How much 'better' is good enough? The magnitude of treatment effect in clinical trials

Abstract

Objectives: Among the various factors required to calculate sample size for clinical trials, the magnitude of treatment effect anticipated is an important component. The objective of this report is to present some of the complexities involved in selection of treatment effect size in clinical trials. As a framework for discussion, an analysis of published reports related to surfactant therapy was carried out.

Design: Twenty-one consecutive exogenous surfactant trials for neonatal respiratory distress syndrome were analyzed. The "Methods" sections were reviewed for evaluating various components of sample size calculation, including the anticipated treatment effect size.

Results: Sixteen (76%) of the 21 reports provided a description of sample size calculations, and 12 of these gave some reasons for the choice of the anticipated treatment effect size. Expressed as percent change, the median treatment effect from intervention anticipated by the investigators was 50% (range, 15% to 90%), with a positively skewed distribution. The actual median percent reduction in adverse events from treatment (as compared with baseline) was 36% (range, 75% reduction to 5% excess). When the treatment effect was expressed as difference in adverse event rate, in the 14 (of 16) trials that could be analyzed, the median observed reduction in adverse events (death, bronchopulmonary dysplasia, or occurrence of respiratory distress syndrome) was 14.5% (range, 52% reduction to 2% excess). All trials except one concluded, however, that the intervention was effective, mostly based on additional subgroup calculations.

Conclusions: Researchers often select sample sizes capable of detecting only large treatment effects, thus risking type II error, although sometimes a much smaller effect could be clinically important. While pragmatic considerations must be considered during the design of randomized clinical trials, researchers ought to present a rationale for anticipating a given magnitude of treatment effect in their sample size calculations. It may be possible to consider innovative trial designs that help determine the most appropriate treatment choice with the least possible sample size.