High replicability of newly discovered social-behavioural findings is achievable

John Protzko^{1

2}, Jon Krosnick³, Leif Nelson⁴, Brian A Nosek^{5

6}, Jordan Axt⁷, Matt Berent⁸, Nicholas Buttrick⁹, Matthew DeBell³, Charles R Ebersole⁶, Sebastian Lundmark¹⁰, Bo MacInnis³, Michael O'Donnell¹¹, Hannah Perfecto¹², James E Pustejovsky¹³, Scott S Roeder¹⁴, Jan Walleczek¹⁵, Jonathan W Schooler¹⁶

Affiliations

¹ Department of Psychological & Brain Sciences, University of California, Santa Barbara, Santa Barbara, CA, USA. protzko@gmail.com.
² Department of Psychological Science, Central Connecticut State University, New Britain, CT, USA. protzko@gmail.com.
³ Institute for Research in the Social Sciences, Stanford University, Stanford, CA, USA.
⁴ Haas School of Business, University of California, Berkeley, Berkeley, CA, USA.
⁵ Center for Open Science, Charlottesville, VA, USA.
⁶ Department of Psychology, University of Virginia, Charlottesville, VA, USA.
⁷ Department of Psychology, McGill University, Montreal, Quebec, Canada.
⁸ Matt Berent Consulting, Sharon, PA, USA.
⁹ Department of Psychology, University of Wisconsin-Madison, Madison, WI, USA.
¹⁰ SOM Institute, University of Gothenburg, Gothenburg, Sweden.
¹¹ McDonough School of Business, Georgetown University, Washington, DC, USA.
¹² Olin School of Business, Washington University in St. Louis, St. Louis, MO, USA.
¹³ Educational Psychology Department, University of Wisconsin-Madison, Madison, WI, USA.
¹⁴ Darla Moore School of Business, University of South Carolina, Columbia, SC, USA.
¹⁵ Phenoscience Laboratories, Berlin, Germany.
¹⁶ Department of Psychological & Brain Sciences, University of California, Santa Barbara, Santa Barbara, CA, USA.

PMID: 37945809
PMCID: PMC10896719
DOI: 10.1038/s41562-023-01749-9

High replicability of newly discovered social-behavioural findings is achievable

John Protzko et al. Nat Hum Behav. 2024 Feb.

. 2024 Feb;8(2):311-319.

doi: 10.1038/s41562-023-01749-9. Epub 2023 Nov 9.

Authors

Affiliations

¹ Department of Psychological & Brain Sciences, University of California, Santa Barbara, Santa Barbara, CA, USA. protzko@gmail.com.
² Department of Psychological Science, Central Connecticut State University, New Britain, CT, USA. protzko@gmail.com.
³ Institute for Research in the Social Sciences, Stanford University, Stanford, CA, USA.
⁴ Haas School of Business, University of California, Berkeley, Berkeley, CA, USA.
⁵ Center for Open Science, Charlottesville, VA, USA.
⁶ Department of Psychology, University of Virginia, Charlottesville, VA, USA.
⁷ Department of Psychology, McGill University, Montreal, Quebec, Canada.
⁸ Matt Berent Consulting, Sharon, PA, USA.
⁹ Department of Psychology, University of Wisconsin-Madison, Madison, WI, USA.
¹⁰ SOM Institute, University of Gothenburg, Gothenburg, Sweden.
¹¹ McDonough School of Business, Georgetown University, Washington, DC, USA.
¹² Olin School of Business, Washington University in St. Louis, St. Louis, MO, USA.
¹³ Educational Psychology Department, University of Wisconsin-Madison, Madison, WI, USA.
¹⁴ Darla Moore School of Business, University of South Carolina, Columbia, SC, USA.
¹⁵ Phenoscience Laboratories, Berlin, Germany.
¹⁶ Department of Psychological & Brain Sciences, University of California, Santa Barbara, Santa Barbara, CA, USA.

PMID: 37945809
PMCID: PMC10896719
DOI: 10.1038/s41562-023-01749-9

Retraction in

Retraction Note: High replicability of newly discovered social-behavioural findings is achievable.
Protzko J, Krosnick J, Nelson L, Nosek BA, Axt J, Berent M, Buttrick N, DeBell M, Ebersole CR, Lundmark S, MacInnis B, O'Donnell M, Perfecto H, Pustejovsky JE, Roeder SS, Walleczek J, Schooler JW. Protzko J, et al. Nat Hum Behav. 2024 Oct;8(10):2067. doi: 10.1038/s41562-024-01997-3. Nat Hum Behav. 2024. PMID: 39317794 Free PMC article. No abstract available.

Abstract

Failures to replicate evidence of new discoveries have forced scientists to ask whether this unreliability is due to suboptimal implementation of methods or whether presumptively optimal methods are not, in fact, optimal. This paper reports an investigation by four coordinated laboratories of the prospective replicability of 16 novel experimental findings using rigour-enhancing practices: confirmatory tests, large sample sizes, preregistration and methodological transparency. In contrast to past systematic replication efforts that reported replication rates averaging 50%, replication attempts here produced the expected effects with significance testing (P < 0.05) in 86% of attempts, slightly exceeding the maximum expected replicability based on observed effect sizes and sample sizes. When one lab attempted to replicate an effect discovered by another lab, the effect size in the replications was 97% that in the original study. This high replication rate justifies confidence in rigour-enhancing methods to increase the replicability of new discoveries.

PubMed Disclaimer

Conflict of interest statement

B.A.N. is the executive director of the non-profit Center for Open Science with a mission to “increase openness, integrity, and reproducibility of research”. J.W. was the scientific director of the Fetzer Franklin Fund that sponsored this research, and B.A.N. and J.W.S were also on the scientific advisory board. J.W. made substantive contributions to the design and execution of this research, but as a funder did not have controlling interest in the decision to publish or not. All other authors declare no competing interests.

Figures

**Fig. 1. Effect size estimates and 95% CI from 16 new discoveries in the social-behavioral sciences, with four replications each.**
ESs (shapes) and 95% CIs from 16 new discoveries (yellow) in the social-behavioural sciences with four replications each. Each lab is designated by a unique shape for the observed ES; blue marks correspond to self-replications, green marks to independent replications.

Fig. 2. Difference in effect size estimates between self- and independent replications for 16 new discoveries in the social-behavioural sciences, compared with initial confirmation study effect size estimates.
Difference in ESs (shapes) between self-replications and independent replications for the 16 discoveries, compared with the self-confirmatory test ES (0.0 on the x axis). The independent replication estimates are fixed-effects meta-analytic estimates of the three independent studies. The meta-analytic estimates with prediction intervals in the bottom panel combine across all 16 discoveries.

**Fig. 3. Changes in effect size across replications, with initial confirmation study at the intercept.**
Slopes of ES changes across replications, with the self-confirmatory test as the intercept.

See this image and copyright information in PMC

References

1. Putnam, H. Mathematics, Matter and Method: Volume 1, Philosophical Papers (CUP Archive, 1975).
1. Schmidt, S. Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Rev. Gen. Psychol.13, 90–100 (2009).
1. Begley, C. G. & Ellis, L. M. Drug development: raise standards for preclinical cancer research. Nature483, 531–533 (2012). - PubMed
1. Camerer, C. F. et al. Evaluating replicability of laboratory experiments in economics. Science351, 1433–1436 (2016). - PubMed
1. Camerer, C. F. et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat. Hum. Behav.2, 637–644 (2018). - PubMed

Publication types

Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions

LinkOut - more resources

Full Text Sources
Miscellaneous
- NCI CPTAC Assay Portal

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Retracted article

High replicability of newly discovered social-behavioural findings is achievable

Affiliations

High replicability of newly discovered social-behavioural findings is achievable

Authors

Affiliations

Retraction in

Abstract

Conflict of interest statement

Figures

References

Publication types

MeSH terms

LinkOut - more resources

Full Text Sources

Miscellaneous