Open research practices: unintended consequences and suggestions for averting them. (Commentary on the Peer Reviewers' Openness Initiative)

Abstract

The Peer Reviewers' Openness Initiative (PROI) is a move to enlist reviewers in the promotion of data-sharing. In this commentary, I discuss objections that can be raised, first to the specific proposals in the PROI, and second to data-sharing in general. I argue that although many objections have strong counter-arguments, others merit more serious consideration. Regarding the PROI, I suggest that it could backfire if editors and authors feel coerced into data-sharing and so may not be the most pragmatic way of encouraging greater openness. More generally, while promoting data-sharing, we need to be sensitive to cases where sharing of data from human participants could create ethical problems. Furthermore, those interested in promoting reproducible science need to defend against an increased risk of data-dredging when large, multivariable datasets are shared. I end with some suggestions to avoid these unintended consequences.

Keywords: data-dredging; data-sharing; ethics; reproducibility.

Figure 1.

The Garden of Forking Paths: an illustration of how selection of specific measures or subgroups can increase the number of possible comparisons, affecting the likelihood of obtaining a ‘significant’ p-value by chance. See text for explanation, or refer https://figshare.com/s/f50547936de09afef9ff for animated version. The figure title is inspired by Gelman & Loken's [13] literary reference. O, Old; Y, Young; S, Hand skill; P, Hand preference; M, male; F, female; U, urban; R, rural.