Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Jan;17(1):239-251.
doi: 10.1177/1745691620979806. Epub 2021 Mar 8.

Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014-2017)

Tom E Hardwicke  1   2 , Robert T Thibault  3   4 , Jessica E Kosie  5 , Joshua D Wallach  6 , Mallory C Kidwell  7 , John P A Ioannidis  2   8   9
Affiliations

Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014-2017)

Tom E Hardwicke et al. Perspect Psychol Sci. 2022 Jan.

Abstract

Psychologists are navigating an unprecedented period of introspection about the credibility and utility of their discipline. Reform initiatives emphasize the benefits of transparency and reproducibility-related research practices; however, adoption across the psychology literature is unknown. Estimating the prevalence of such practices will help to gauge the collective impact of reform initiatives, track progress over time, and calibrate future efforts. To this end, we manually examined a random sample of 250 psychology articles published between 2014 and 2017. Over half of the articles were publicly available (154/237, 65%, 95% confidence interval [CI] = [59%, 71%]); however, sharing of research materials (26/183; 14%, 95% CI = [10%, 19%]), study protocols (0/188; 0%, 95% CI = [0%, 1%]), raw data (4/188; 2%, 95% CI = [1%, 4%]), and analysis scripts (1/188; 1%, 95% CI = [0%, 1%]) was rare. Preregistration was also uncommon (5/188; 3%, 95% CI = [1%, 5%]). Many articles included a funding disclosure statement (142/228; 62%, 95% CI = [56%, 69%]), but conflict-of-interest statements were less common (88/228; 39%, 95% CI = [32%, 45%]). Replication studies were rare (10/188; 5%, 95% CI = [3%, 8%]), and few studies were included in systematic reviews (21/183; 11%, 95% CI = [8%, 16%]) or meta-analyses (12/183; 7%, 95% CI = [4%, 10%]). Overall, the results suggest that transparency and reproducibility-related research practices were far from routine. These findings establish baseline prevalence estimates against which future progress toward increasing the credibility and utility of psychology research can be compared.

Keywords: meta-research; open science; psychology; reproducibility; transparency.

PubMed Disclaimer

Conflict of interest statement

Declaration of Conflicting Interests: The author(s) declared that there were no conflicts of interest with respect to the authorship or the publication of this article.

Figures

Fig. 1.
Fig. 1.
Assessment of transparency and reproducibility-related research practices in psychology. The x-axis shows the percentage of the total number of articles (N) assessed for a given indicator (which was contingent on the study design; see Table 1). Raw counts are shown inside bars.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

  • Methodological issues in assessing job stress and burnout in psychosocial research.
    Campanini P. Campanini P. Med Lav. 2021 Aug 26;112(4):264-267. doi: 10.23749/mdl.v112i4.12112. Med Lav. 2021. PMID: 34446681 Free PMC article.
  • Open Science Practices in Gambling Research Publications (2016-2019): A Scoping Review.
    Louderback ER, Gainsbury SM, Heirene RM, Amichia K, Grossman A, Bernhard BJ, LaPlante DA. Louderback ER, et al. J Gambl Stud. 2023 Jun;39(2):987-1011. doi: 10.1007/s10899-022-10120-y. Epub 2022 Jun 9. J Gambl Stud. 2023. PMID: 35678905 Free PMC article.
  • Correcting for outcome reporting bias in a meta-analysis: A meta-regression approach.
    van Aert RCM, Wicherts JM. van Aert RCM, et al. Behav Res Methods. 2024 Mar;56(3):1994-2012. doi: 10.3758/s13428-023-02132-2. Epub 2023 Jul 24. Behav Res Methods. 2024. PMID: 37540470 Free PMC article.
  • Teaching open and reproducible scholarship: a critical review of the evidence base for current pedagogical methods and their outcomes.
    Pownall M, Azevedo F, König LM, Slack HR, Evans TR, Flack Z, Grinschgl S, Elsherif MM, Gilligan-Lee KA, de Oliveira CMF, Gjoneska B, Kalandadze T, Button K, Ashcroft-Jones S, Terry J, Albayrak-Aydemir N, Děchtěrenko F, Alzahawi S, Baker BJ, Pittelkow MM, Riedl L, Schmidt K, Pennington CR, Shaw JJ, Lüke T, Makel MC, Hartmann H, Zaneva M, Walker D, Verheyen S, Cox D, Mattschey J, Gallagher-Mitchell T, Branney P, Weisberg Y, Izydorczak K, Al-Hoorie AH, Creaven AM, Stewart SLK, Krautter K, Matvienko-Sikar K, Westwood SJ, Arriaga P, Liu M, Baum MA, Wingen T, Ross RM, O'Mahony A, Bochynska A, Jamieson M, Tromp MV, Yeung SK, Vasilev MR, Gourdon-Kanhukamwe A, Micheli L, Konkol M, Moreau D, Bartlett JE, Clark K, Brekelmans G, Gkinopoulos T, Tyler SL, Röer JP, Ilchovska ZG, Madan CR, Robertson O, Iley BJ, Guay S, Sladekova M, Sadhwani S; FORRT. Pownall M, et al. R Soc Open Sci. 2023 May 17;10(5):221255. doi: 10.1098/rsos.221255. eCollection 2023 May. R Soc Open Sci. 2023. PMID: 37206965 Free PMC article. Review.
  • Eleven strategies for making reproducible research and open science training the norm at research institutions.
    Kohrs FE, Auer S, Bannach-Brown A, Fiedler S, Haven TL, Heise V, Holman C, Azevedo F, Bernard R, Bleier A, Bössel N, Cahill BP, Castro LJ, Ehrenhofer A, Eichel K, Frank M, Frick C, Friese M, Gärtner A, Gierend K, Grüning DJ, Hahn L, Hülsemann M, Ihle M, Illius S, König L, König M, Kulke L, Kutlin A, Lammers F, Mehler DMA, Miehl C, Müller-Alcazar A, Neuendorf C, Niemeyer H, Pargent F, Peikert A, Pfeuffer CU, Reinecke R, Röer JP, Rohmann JL, Sánchez-Tójar A, Scherbaum S, Sixtus E, Spitzer L, Straßburger VM, Weber M, Whitmire CJ, Zerna J, Zorbek D, Zumstein P, Weissgerber TL. Kohrs FE, et al. Elife. 2023 Nov 23;12:e89736. doi: 10.7554/eLife.89736. Elife. 2023. PMID: 37994903 Free PMC article.
See all "Cited by" articles

References

    1. Alsheikh-Ali A. A., Qureshi W., Al-Mallah M. H., Ioannidis J. P. A. (2011). Public availability of published research data in high-impact journals. PLOS ONE, 6(9), Article e24357. 10.1371/journal.pone.0024357 - DOI - PMC - PubMed
    1. Anderson M. S., Ronning E. A., Devries R., Martinson B. C. (2010). Extending the Mertonian norms: Scientists’ subscription to norms of research. The Journal of Higher Education, 81(3), 366–393. 10.1353/jhe.0.0095 - DOI - PMC - PubMed
    1. Baker M. (2016). 1,500 scientists lift the lid on reproducibility. Nature, 533(7604), 452–454. 10.1038/533452a - DOI - PubMed
    1. Bakker M., Veldkamp C. L. S., van Assen M. A. L. M., Crompvoets E. A. V., Ong H. H., Nosek B. A., Soderberg C. K., Mellor D., Wicherts J. M. (2020). Ensuring the quality and specificity of preregistrations. PLOS Biology, 18(12), Article e3000937. 10.1371/journal.pbio.3000937 - DOI - PMC - PubMed
    1. Bekelman J. E., Li Y., Gross C. P. (2003). Scope and impact of financial conflicts of interest in biomedical research: A systematic review. JAMA, 289(4), 454–465. 10.1001/jama.289.4.454 - DOI - PubMed

Publication types

LinkOut - more resources