A guide for authors and readers of the American Society for Nutrition Journals on the proper use of P values and strategies that promote transparency and improve research reproducibility

Abstract

Two questions regarding the scientific literature have become grist for public discussion: 1) what place should P values have in reporting the results of studies? 2) How should the perceived difficulty in replicating the results reported in published studies be addressed? We consider these questions to be 2 sides of the same coin; failing to address them can lead to an incomplete or incorrect message being sent to the reader. If P values (which are derived from the estimate of the effect size and a measure of the precision of the estimate of the effect) are used improperly, for example reporting only significant findings, or reporting P values without account for multiple comparisons, or failing to indicate the number of tests performed, the scientific record can be biased. Moreover, if there is a lack of transparency in the conduct of a study and reporting of study results, it will not be possible to repeat a study in a manner that allows inferences from the original study to be reproduced or to design and conduct a different experiment whose aim is to confirm the original study's findings. The goal of this article is to discuss how P values can be used in a manner that is consistent with the scientific method, and to increase transparency and reproducibility in the conduct and analysis of nutrition research.

Keywords: P value; reliability; reproducibility; strategies; transparency.

FIGURE 1

Relation between sample size and P value of an analysis from a 2-sample paired Student's t-test. The mean of 1 group was 0.0, the mean of the second group was 0.18. The SD of both groups was 1. Each estimate represents the mean of 10,000 simulations. The sample sizes for each group, experimental and control, in each of the simulations was 100, 125, 167, 250, 333, 500, 667, 833, and 1000 subjects. The horizontal dashed line represents the “standard” 2-tailed P value (0.05) used to establish significance.

FIGURE 2

The probability of ≥1 false-positive result from a series of tests (the number of tests appears on the abscissa) when the null hypothesis, H0 is true (i.e. when there is no difference between experimental and control groups). Each line represents a series of tests with a different significance (i.e. type-I error rate), 0.0005, 0.001, 0.01, 0.025, and 0.05. The points of the lines were calculated using the formula probability = 1−(1 − Type-I Error rate)^{Number of Tests}. See supplementary material for a derivation of the formula.