The significance of differences between means. An empirical study

Abstract

Three tests of statistical significance: the confidence interval (C), Student's t, and the variance ratio, (F) were applied to 13 homologous sets of 26 differences each between means of groups of six. The C test is shown to be most reliable in separating two distinct populations of values, as significant or not significant, at the 1% level of probability of a larger difference. The other tests are less reliable in that neither makes a distinct separation in these terms. The t test is the least rigorous, and the F test the most, in terms of proportion of total number of differences judged significant by these tests, and of a separation of distinct populations of values in terms of magnitude of the difference.