Random assignment of available cases: bootstrap standard errors and confidence intervals

Abstract

A frequently used experimental design in psychological research randomly divides a set of available cases, a local population, between 2 treatments and then applies an independent-samples t test to either test a hypothesis about or estimate a confidence interval (CI) for the population mean difference in treatment response. C. S. Reichardt and H. F. Gollob (1999) established that the t test can be conservative for this design-yielding hypothesis test P values that are too large or CIs that are too wide for the relevant local population. This article develops a less conservative approach to local population inference, one based on the logic of B. Efron's (1979) nonparametric bootstrap. The resulting randomization bootstrap is then compared with an established approach to local population inference, that based on randomization or permutation tests. Finally, the importance of local population inference is established by reference to the distinction between statistical and scientific inference.