Open science interventions to improve reproducibility and replicability of research: a scoping review

Abstract

Various open science practices have been proposed to improve the reproducibility and replicability of scientific research, but not for all practices, there may be evidence they are indeed effective. Therefore, we conducted a scoping review of the literature on interventions to improve reproducibility. We systematically searched Medline, Embase, Web of Science, PsycINFO, Scopus and Eric, on 18 August 2023. Any study empirically evaluating the effectiveness of interventions aimed at improving the reproducibility or replicability of scientific methods and findings was included. We summarized the retrieved evidence narratively and in evidence gap maps. Of the 105 distinct studies we included, 15 directly measured the effect of an intervention on reproducibility or replicability, while the remainder addressed a proxy outcome that might be expected to increase reproducibility or replicability, such as data sharing, methods transparency or pre-registration. Thirty studies were non-comparative and 27 were comparative but cross-sectional observational designs, precluding any causal inference. Despite studies investigating a range of interventions and addressing various outcomes, our findings indicate that in general the evidence base for which various interventions to improve reproducibility of research remains remarkably limited in many respects.

Keywords: open science; reproducibility; scoping review.

Figure 1.

PRISMA flowchart of the search and selection process. The boxes show the numbers of retrieved, excluded and sifted articles in each step of the searching, screening and selection process. In the end, we included 86 articles, containing 105 studies.

Figure 2.

Evidence gap map of interventions and specific outcomes investigated, with information on study designs. Study designs: between = comparative (between-subject comparison); within = comparative (within-subject comparison/repeated measures design); post only = post-intervention (only a post-measurement after the implementation of an intervention and the intervention is explicitly mentioned); qualitative = a qualitative description of what happened during the implementation of an intervention; other = other designs. The size of the bubbles refers to the number of studies that address this particular intervention/outcome combination; the colours refer to different study designs.

Figure 3.

Evidence gap map of interventions and outcome domains investigated, with information on study designs. The left pane shows direct reproducibility outcomes, while the right pane shows proxy outcomes. Study designs: between = comparative (between-subject comparison); within = comparative (within-subject comparison/repeated measures design); post only = post-intervention (only a post-measurement after the implementation of an intervention and the intervention is explicitly mentioned); qualitative = a qualitative description of what happened during the implementation of an intervention; other = other designs. The size of the bubbles refers to the number of studies that address this particular intervention/outcome combination; the colours refer to different study designs.

Figure 4.

Evidence gap map of interventions and specific outcomes investigated, with information on author-stated effects. Judgment of the effect of the intervention by the authors of the study. Green = generally positive; pink = generally negative; grey = null/neutral. The size of the bubbles was defined by the number of studies that address this particular intervention/outcome combination.

Figure 5.

Disciplinary distribution and temporal trends of included intervention studies. Disciplines were coded according to the Frascati manual discipline ’Fields of Science and Technology’ [125]. A. Number of studies according to Frascati Field of Science and Technology Classification schema top-level ‘Disciplines’. B. Number of studies according to Frascati second-level ‘Knowledge Fields’, with respective top-level categories indicated by colour. C. Number of studies over time, with respective top-level categories indicated by colour. Note that each study may cover more than one discipline/field of knowledge. In these cases, the studies are counted fully for each discipline/field of knowledge, and hence overall numbers add up to more than our included number of studies.