Better Inference in Neuroscience: Test Less, Estimate More

Abstract

Null-hypothesis significance testing (NHST) has become the main tool of inference in neuroscience, and yet evidence suggests we do not use this tool well: tests are often planned poorly, conducted unfairly, and interpreted invalidly. This editorial makes the case that in addition to reforms to increase rigor we should test less, reserving NHST for clearly confirmatory contexts in which the researcher has derived a quantitative prediction, can provide the inputs needed to plan a quality test, and can specify the criteria not only for confirming their hypothesis but also for rejecting it. A reduction in testing would be accompanied by an expansion of the use of estimation [effect sizes and confidence intervals (CIs)]. Estimation is more suitable for exploratory research, provides the inputs needed to plan strong tests, and provides important contexts for properly interpreting tests.

Figure 1.

Strong testing via inference by interval. In this approach to testing, the researcher defines the range of effect sizes that are negligible relative to the research question at hand (gray box with dashed border). Significance at α = 0.05 can then be seen graphically by plotting the effect size in the sample (triangle) with the 95% confidence interval (thin bar) and 90% confidence interval (thick bar). The effect is deemed substantive if the entire 95% confidence interval is outside the zone of negligible effects, negligible if the entire 90% confidence interval is inside this zone, and ambiguous otherwise.