Assessing computational reproducibility in Behavior Research Methods

Abstract

Psychological science has thrived thanks to new methods and innovative practices. Journals, including Behavior Research Methods (BRM), continue to support the dissemination and evaluation of research assets including data, software/hardware, statistical code, and databases of stimuli. However, such research assets rarely allow for computational reproducibility, meaning they are difficult to reuse. Therefore, in this preregistered report, we explore how BRM's authors and BRM structures shape the landscape of functional research assets. Our broad research questions concern: (1) How quickly methods and analytical techniques reported in BRM can be used and developed further by other scientists; (2) Whether functionality has improved following changes to BRM journal policy in support of computational reproducibility; (3) Whether we can disentangle such policy changes from changes in reproducibility over time. We randomly sampled equal numbers of papers (N = 204) published in BRM before and after the implementation of policy changes. Pairs of researchers recorded how long it took to ensure assets (data, software/hardware, statistical code, and materials) were fully operational. They also coded the completeness and reusability of the assets. While improvements were observed in all measures, only changes to completeness were altered significantly following the policy changes (d = .37). The effects varied between different types of research assets, with data sets from surveys/experiments showing the largest improvements in completeness and reusability. Perhaps more importantly, changes to policy do appear to have improved the life span of research products by reducing natural decline. We conclude with a discussion of how, in the future, research and policy might better support computational reproducibility within and beyond psychological science.

Keywords: Meta-research; Open science; Reproducibility; Research policy; Transparency.

Fig. 1

Counts by primary research asset showing the split between papers before and after journal policy changes; 61.5% of the sample was assigned a second category (for which the order of most to least common asset remained the same as shown here). A breakdown of which research assets were coded by each pair is reported in the Supplementary Materials (Table A)

Fig. 2

Distributions of time (a), completeness (b), and reusability (c) across the whole sample (N = 200)

Fig. 3

Distributions of time (a), completeness (b), and reusability (c) before and after changes to journal policy

Fig. 4

Average time (a), completeness (b), and reusability (c) among primary research assets before and after changes to journal policy. Error bars illustrate standard error. A breakdown of how assets were coded by each pair is reported in the Supplementary Materials (Fig. A)

Fig. 5

Measures of time (a), completeness (b), and reusability (c) plotted against time since publication for pre-policy articles (‘Before’) and current policy articles (‘After’). While higher values for completeness and reusability are proxies for improved quality, the direction of decline is reversed for time, where lower values are preferable (meaning assets were quicker to access and use). The red dotted line indicates where the policy change occurred: the majority of current policy articles (to the left) would have been submitted after January 1, 2020. See Fig. B in the Supplementary Materials for comparisons between research assets