How can we make sound replication decisions?

doi:10.1073/pnas.2401236121

. 2025 Feb 4;122(5):e2401236121.

doi: 10.1073/pnas.2401236121. Epub 2025 Jan 27.

How can we make sound replication decisions?

Clintin P Davis-Stober^#^{1

2}, Alexandra Sarafoglou^#³, Balazs Aczel⁴, Suyog H Chandramouli^{5

6}, Timothy M Errington⁷, Sarahanne M Field⁸, Ayelet Fishbach⁹, Juliana Freire^{10

11}, John P A Ioannidis^{12

13

14

15}, Klaus Oberauer¹⁶, Franco Pestilli^{17

18

19}, Susanne Ressl¹⁸, Daniel J Schad²⁰, Judith Ter Schure²¹, Katya Tentori²², Don van Ravenzwaaij²³, Joachim Vandekerckhove^{24

25

26}, Odd Erik Gundersen²⁷

Affiliations

¹ Department of Psychological Sciences, University of Missouri, Columbia, MO 65211.
² University of Missouri Institute for Data Science and Informatics, University of Missouri, Columbia, MO 65211.
³ Department of Psychology, Psychological Methods Unit, University of Amsterdam, Amsterdam 1001 NK, The Netherlands.
⁴ Department of Affective Psychology, Institute of Psychology, Eotvos Lorand University, Budapest 1063, Hungary.
⁵ Department of Information and Computer Engineering, Aalto University, Espoo 02150, Finland.
⁶ Department of Psychology, Princeton University, Princeton, NJ 08544.
⁷ Center for Open Science, Charlottesville, VA 22903.
⁸ Pedagogical and Educational Sciences, University of Groningen, Groningen 9712 TJ, The Netherlands.
⁹ Booth School of Business, University of Chicago, Chicago, IL 60637.
¹⁰ Department of Computer Science, Tandon School of Engineering, New York University, New York, NY 10011.
¹¹ Center for Data Science, New York University, New York, NY 10011.
¹² Department of Medicine, Stanford University, Stanford, CA 94305.
¹³ Department of Epidemiology and Population Health, Stanford University, Stanford, CA 94305.
¹⁴ Department of Biomedical Data Science, Stanford University, Stanford, CA 94305.
¹⁵ Department of Meta-Research Innovation Center at Stanford, Stanford University, Stanford, CA 94305.
¹⁶ Department of Psychology, University of Zurich, Zurich 8050, Switzerland.
¹⁷ Department of Psychology, University of Texas at Austin, Austin, TX 78712.
¹⁸ Department of Neuroscience, University of Texas at Austin, Austin, TX 78712.
¹⁹ Center for Learning and Memory, The University of Texas at Austin, Austin, TX 78712.
²⁰ Institute of Mind, Brain and Behavior, Psychology Department, Health and Medical University, Potsdam 14471, Germany.
²¹ Department of Epidemiology and Data Science, Amsterdam University Medical Center, Amsterdam 1105AZ, The Netherlands.
²² Center for Mind/Brain Sciences, University of Trento, Rovereto 38068, Italy.
²³ Department of Psychology, Psychometrics and Statistics, University of Groningen, Groningen 9712 TS, The Netherlands.
²⁴ Department of Cognitive Sciences, University of California, Irvine, CA 92697.
²⁵ Department of Statistics, University of California, Irvine, CA 92697.
²⁶ Department of Logic & Philosophy, University of California, Irvine, CA 92697.
²⁷ Department of Computer Science, Faculty of Information Technology and Electrical Engineering, Norwegian University of Science and Technology, Trondheim 7030, Norway.

^# Contributed equally.

PMID: 39869811
PMCID: PMC11804638
DOI: 10.1073/pnas.2401236121

How can we make sound replication decisions?

Clintin P Davis-Stober et al. Proc Natl Acad Sci U S A. 2025.

. 2025 Feb 4;122(5):e2401236121.

doi: 10.1073/pnas.2401236121. Epub 2025 Jan 27.

Authors

Affiliations

¹ Department of Psychological Sciences, University of Missouri, Columbia, MO 65211.
² University of Missouri Institute for Data Science and Informatics, University of Missouri, Columbia, MO 65211.
³ Department of Psychology, Psychological Methods Unit, University of Amsterdam, Amsterdam 1001 NK, The Netherlands.
⁴ Department of Affective Psychology, Institute of Psychology, Eotvos Lorand University, Budapest 1063, Hungary.
⁵ Department of Information and Computer Engineering, Aalto University, Espoo 02150, Finland.
⁶ Department of Psychology, Princeton University, Princeton, NJ 08544.
⁷ Center for Open Science, Charlottesville, VA 22903.
⁸ Pedagogical and Educational Sciences, University of Groningen, Groningen 9712 TJ, The Netherlands.
⁹ Booth School of Business, University of Chicago, Chicago, IL 60637.
¹⁰ Department of Computer Science, Tandon School of Engineering, New York University, New York, NY 10011.
¹¹ Center for Data Science, New York University, New York, NY 10011.
¹² Department of Medicine, Stanford University, Stanford, CA 94305.
¹³ Department of Epidemiology and Population Health, Stanford University, Stanford, CA 94305.
¹⁴ Department of Biomedical Data Science, Stanford University, Stanford, CA 94305.
¹⁵ Department of Meta-Research Innovation Center at Stanford, Stanford University, Stanford, CA 94305.
¹⁶ Department of Psychology, University of Zurich, Zurich 8050, Switzerland.
¹⁷ Department of Psychology, University of Texas at Austin, Austin, TX 78712.
¹⁸ Department of Neuroscience, University of Texas at Austin, Austin, TX 78712.
¹⁹ Center for Learning and Memory, The University of Texas at Austin, Austin, TX 78712.
²⁰ Institute of Mind, Brain and Behavior, Psychology Department, Health and Medical University, Potsdam 14471, Germany.
²¹ Department of Epidemiology and Data Science, Amsterdam University Medical Center, Amsterdam 1105AZ, The Netherlands.
²² Center for Mind/Brain Sciences, University of Trento, Rovereto 38068, Italy.
²³ Department of Psychology, Psychometrics and Statistics, University of Groningen, Groningen 9712 TS, The Netherlands.
²⁴ Department of Cognitive Sciences, University of California, Irvine, CA 92697.
²⁵ Department of Statistics, University of California, Irvine, CA 92697.
²⁶ Department of Logic & Philosophy, University of California, Irvine, CA 92697.
²⁷ Department of Computer Science, Faculty of Information Technology and Electrical Engineering, Norwegian University of Science and Technology, Trondheim 7030, Norway.

^# Contributed equally.

PMID: 39869811
PMCID: PMC11804638
DOI: 10.1073/pnas.2401236121

Abstract

Replication and the reported crises impacting many fields of research have become a focal point for the sciences. This has led to reforms in publishing, methodological design and reporting, and increased numbers of experimental replications coordinated across many laboratories. While replication is rightly considered an indispensable tool of science, financial resources and researchers' time are quite limited. In this perspective, we examine different values and attitudes that scientists can consider when deciding whether to replicate a finding and how. We offer a conceptual framework for assessing the usefulness of various replication tools, such as preregistration.

Keywords: methodology; reform; replication; reproducibility.

PubMed Disclaimer

Conflict of interest statement

Competing interests statement:The authors declare no competing interest.

Figures

**Fig. 1.**
This is an illustration of the conceptual framework we used to guide our discussion. Values, both nonepistemic and epistemic, influence decisions about replication, which are, in turn, weighed via the cognitive attitudes of the scientist making those decisions.

See this image and copyright information in PMC

Cited by

Discourse on measurement.
Pedersen AP, Kellen D, Mayo-Wilson C, Davis-Stober CP, Dunn JC, Khan MA, Stinchcombe MB, Kalish ML, Tentori K, Haaf J. Pedersen AP, et al. Proc Natl Acad Sci U S A. 2025 Feb 4;122(5):e2401229121. doi: 10.1073/pnas.2401229121. Epub 2025 Jan 27. Proc Natl Acad Sci U S A. 2025. PMID: 39869800 Free PMC article.
Dialogues about the practice of science.
Shiffrin RM, Trueblood JS, Kellen D, Vandekerckhove J. Shiffrin RM, et al. Proc Natl Acad Sci U S A. 2025 Feb 4;122(5):e2423782122. doi: 10.1073/pnas.2423782122. Epub 2025 Jan 27. Proc Natl Acad Sci U S A. 2025. PMID: 39869805 Free PMC article. No abstract available.

References

1. Ioannidis J. P., Contradicted and initially stronger effects in highly cited clinical research. Jama 294, 218–228 (2005). - PubMed
1. Nosek B. A., et al. , Replicability, robustness, and reproducibility in psychological science. Annu. Rev. Psychol. 73, 719–748 (2022). - PubMed
1. Pashler H., Wagenmakers E. J., Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspect. Psychol. Sci. 7, 528–530 (2012). - PubMed
1. Camerer C. F., et al. , Evaluating replicability of laboratory experiments in economics. Science 351, 1433–1436 (2016). - PubMed
1. Cockburn A., Dragicevic P., Besançon L., Gutwin C., Threats of a replication crisis in empirical computer science. Commun. ACM 63, 70–79 (2020).

MeSH terms

Actions
Actions
Actions
Actions

Grants and funding

LinkOut - more resources

Full Text Sources

[1] Ioannidis J. P., Contradicted and initially stronger effects in highly cited clinical research. Jama 294, 218–228 (2005). - PubMed

[2] Ioannidis J. P., Contradicted and initially stronger effects in highly cited clinical research. Jama 294, 218–228 (2005). - PubMed

[3] Nosek B. A., et al. , Replicability, robustness, and reproducibility in psychological science. Annu. Rev. Psychol. 73, 719–748 (2022). - PubMed

[4] Nosek B. A., et al. , Replicability, robustness, and reproducibility in psychological science. Annu. Rev. Psychol. 73, 719–748 (2022). - PubMed

[5] Pashler H., Wagenmakers E. J., Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspect. Psychol. Sci. 7, 528–530 (2012). - PubMed

[6] Pashler H., Wagenmakers E. J., Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspect. Psychol. Sci. 7, 528–530 (2012). - PubMed

[7] Camerer C. F., et al. , Evaluating replicability of laboratory experiments in economics. Science 351, 1433–1436 (2016). - PubMed

[8] Camerer C. F., et al. , Evaluating replicability of laboratory experiments in economics. Science 351, 1433–1436 (2016). - PubMed

[9] Cockburn A., Dragicevic P., Besançon L., Gutwin C., Threats of a replication crisis in empirical computer science. Commun. ACM 63, 70–79 (2020).

[10] Cockburn A., Dragicevic P., Besançon L., Gutwin C., Threats of a replication crisis in empirical computer science. Commun. ACM 63, 70–79 (2020).

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

How can we make sound replication decisions?

Affiliations

How can we make sound replication decisions?

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

Similar articles

Cited by

References

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources

Abstract

Conflict of interest statement

Figures

Similar articles

Cited by

References

MeSH terms

Related information

Grants and funding

LinkOut - more resources

Full Text Sources