[Significance, effect size, and confidence interval]

Abstract

The term "statistical significance" is often misunderstood. The result of a study may be labelled to be "highly significant" as if implying "highly important". Statistically significant, however, does only mean that a study result might have been found with a predefined probability (conventionally set at 5 %) even when the null hypothesis is true in the population, i. e. the effect found in the study sample does not exist in reality. Whether a result proves to be significant or not largely depends on sample size. Thus, in a large sample minimal effects of no practical relevance may turn out significant whereas in a small sample even large, important effects may fail to reach the significance level. As a consequence, when presenting the results of a study the effect size should be reported together with a confidence interval indicating the probable range that contains the population effect.