Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results

Abstract

Background: The widespread reluctance to share published research data is often hypothesized to be due to the authors' fear that reanalysis may expose errors in their work or may produce conclusions that contradict their own. However, these hypotheses have not previously been studied systematically.

Methods and findings: We related the reluctance to share research data for reanalysis to 1148 statistically significant results reported in 49 papers published in two major psychology journals. We found the reluctance to share data to be associated with weaker evidence (against the null hypothesis of no effect) and a higher prevalence of apparent errors in the reporting of statistical results. The unwillingness to share data was particularly clear when reporting errors had a bearing on statistical significance.

Conclusions: Our findings on the basis of psychological papers suggest that statistical results are particularly hard to verify when reanalysis is more likely to lead to contrasting conclusions. This highlights the importance of establishing mandatory data archiving policies.

Figure 1. Distribution of reporting errors per paper for papers from which data were shared and from which no data were shared.

Distribution of the number of errors in the reporting of p-values for 28 papers from which the data were not shared (left column) and 21 from which the data were shared (right column) for all misreporting errors (upper row), larger misreporting errors at the 2^nd decimal (middle row), and misreporting errors that concerned statistical significance (p<.05; bottom row).

Figure 2. Distribution of p-values reported as being significant in papers from which data were shared or not.

Distribution of p-values reported as being significant (at p<.05) in 21 papers from which data were shared (N = 561; in black) and in 28 papers from which data were not shared (N = 587; in grey), showing that p-values often lie closer to the typical boundary of significance when data are not shared for reanalysis. Frequencies of reporting errors (as given above the bars) reflect higher error prevalence in papers from which no data were shared.