Error in statistical tests of error in statistical tests

Abstract

Background: A recent paper found that terminal digits of statistical values in Nature deviated significantly from an equiprobable distribution, indicating errors or inconsistencies in rounding. This finding, as well as the discovery that a large percentage of p values were inconsistent with reported test statistics, led to a great deal of concern in the popular press and scientific community. The findings ultimately led to new guidelines for all Nature Research Journals.

Methods: We checked the statistical analysis behind the original paper's tests of equiprobability.

Results: The original paper tested equiprobability with the Kolmogorov-Smirnov test outside its regime of validity. Correct tests find no statistically significant deviations from equiprobability for the statistical values in Nature.

Conclusion: Statistical tests should be used correctly.