The (mis)reporting of statistical results in psychology journals

Abstract

In order to study the prevalence, nature (direction), and causes of reporting errors in psychology, we checked the consistency of reported test statistics, degrees of freedom, and p values in a random sample of high- and low-impact psychology journals. In a second study, we established the generality of reporting errors in a random sample of recent psychological articles. Our results, on the basis of 281 articles, indicate that around 18% of statistical results in the psychological literature are incorrectly reported. Inconsistencies were more common in low-impact journals than in high-impact journals. Moreover, around 15% of the articles contained at least one statistical conclusion that proved, upon recalculation, to be incorrect; that is, recalculation rendered the previously significant result insignificant, or vice versa. These errors were often in line with researchers' expectations. We classified the most common errors and contacted authors to shed light on the origins of the errors.

Fig. 1

Flow chart of articles in Studies 1 and 2

Fig. 2

Overview of the different error categories broken down by high- and low-impact journals and by exactly and inexactly reported statistical results. Wald’s confidence intervals are represented in the figure by the error bars attached to each column

Fig. 3

Erroneously reported p values compared with the recalculated p value. Correctly reported p values would be situated on the diagonal. The dots above the diagonal represent statistical results that were reported as a p value lower than the actual p value. The dots below the diagonal represent statistical results that were reported as a p value higher than the actual p value. The dots in the left upper block represent gross errors in which a nonsignificant result is reported as significant. The dots in the right lower block represent gross errors in which a significant result is reported as nonsignificant