Response to Comment on "Estimating the reproducibility of psychological science"

Christopher J Anderson¹, Štěpán Bahník², Michael Barnett-Cowan³, Frank A Bosco⁴, Jesse Chandler⁵, Christopher R Chartier⁶, Felix Cheung⁷, Cody D Christopherson⁸, Andreas Cordes⁹, Edward J Cremata¹⁰, Nicolas Della Penna¹¹, Vivien Estel¹², Anna Fedor¹³, Stanka A Fitneva¹⁴, Michael C Frank¹⁵, James A Grange¹⁶, Joshua K Hartshorne¹⁷, Fred Hasselman¹⁸, Felix Henninger¹⁹, Marije van der Hulst²⁰, Kai J Jonas²¹, Calvin K Lai²², Carmel A Levitan²³, Jeremy K Miller²⁴, Katherine S Moore²⁵, Johannes M Meixner²⁶, Marcus R Munafò²⁷, Koen I Neijenhuijs²⁸, Gustav Nilsonne²⁹, Brian A Nosek³⁰, Franziska Plessow³¹, Jason M Prenoveau³², Ashley A Ricker³³, Kathleen Schmidt³⁴, Jeffrey R Spies³⁵, Stefan Stieger³⁶, Nina Strohminger³⁷, Gavin B Sullivan³⁸, Robbie C M van Aert³⁹, Marcel A L M van Assen⁴⁰, Wolf Vanpaemel⁴¹, Michelangelo Vianello⁴², Martin Voracek⁴³, Kellylynn Zuni⁴⁴

Affiliations

¹ Russell Sage College, Troy, NY, USA. nosek@virginia.edu.
² University of Würzburg, Würzburg, Germany.
³ University of Waterloo, Waterloo, Ontario, Canada.
⁴ Virginia Commonwealth University, Richmond, VA, USA.
⁵ University of Michigan, Ann Arbor, MI 48104, USA. Mathematica Policy Research, Washington, DC, USA.
⁶ Ashland University, Ashland, OH, USA.
⁷ Michigan State University, East Lansing, MI, USA.
⁸ Southern Oregon University, Ashland, OR, USA.
⁹ University of Göttingen, Institute for Psychology, Göttingen, Germany.
¹⁰ University of Southern California, Los Angeles, CA, USA.
¹¹ Australian National University, Canberra, Australia.
¹² Technische Universität Braunschweig, Braunschweig, Germany.
¹³ Parmenides Stiftung, Munich, Germany.
¹⁴ Queen's University, Kingston, Ontario, Canada.
¹⁵ Stanford University, Stanford, CA, USA.
¹⁶ Keele University, Keele, Staffordshire, UK.
¹⁷ Boston College, Chestnut Hill, MA, USA.
¹⁸ Radboud University Nijmegen, Nijmegen, Netherlands.
¹⁹ University of Koblenz-Landau, Landau, Germany.
²⁰ Erasmus Medical Center, Rotterdam, Netherlands.
²¹ University of Amsterdam, Amsterdam, Netherlands.
²² Harvard University, Cambridge, MA, USA.
²³ Occidental College, Los Angeles, CA, USA.
²⁴ Willamette University, Salem, OR, USA.
²⁵ Arcadia University, Glenside, PA, USA.
²⁶ University of Potsdam, Potsdam, Germany.
²⁷ University of Bristol, Bristol, UK.
²⁸ Vrije Universiteit Amsterdam, Amsterdam, Netherlands.
²⁹ Karolinska Institutet, Stockholm University, Stockholm, Sweden.
³⁰ Center for Open Science, Charlottesville, VA, USA. University of Virginia, Charlottesville, VA, USA. nosek@virginia.edu.
³¹ Harvard Medical School, Boston, MA, USA.
³² Loyola University, Baltimore, MD, USA.
³³ University of California, Riverside, CA, USA.
³⁴ Wesleyan University, Middletown, CT, USA.
³⁵ Center for Open Science, Charlottesville, VA, USA. University of Virginia, Charlottesville, VA, USA.
³⁶ University of Konstanz, Konstanz, Germany.
³⁷ Yale University, New Haven, CT, USA.
³⁸ Coventry University, Coventry, UK.
³⁹ Tilburg University, Tilburg, Netherlands.
⁴⁰ Tilburg University, Tilburg, Netherlands. Utrecht University, Utrecht, Netherlands.
⁴¹ University of Leuven, Leuven, Belgium.
⁴² University of Padova, Padova, Italy.
⁴³ University of Vienna, Vienna, Austria.
⁴⁴ Adams State University, Alamosa, CO, USA.

PMID: 26941312
DOI: 10.1126/science.aad9163

Free article

Comment

Response to Comment on "Estimating the reproducibility of psychological science"

Christopher J Anderson et al. Science. 2016.

Free article

. 2016 Mar 4;351(6277):1037.

doi: 10.1126/science.aad9163.

Authors

Affiliations

¹ Russell Sage College, Troy, NY, USA. nosek@virginia.edu.
² University of Würzburg, Würzburg, Germany.
³ University of Waterloo, Waterloo, Ontario, Canada.
⁴ Virginia Commonwealth University, Richmond, VA, USA.
⁵ University of Michigan, Ann Arbor, MI 48104, USA. Mathematica Policy Research, Washington, DC, USA.
⁶ Ashland University, Ashland, OH, USA.
⁷ Michigan State University, East Lansing, MI, USA.
⁸ Southern Oregon University, Ashland, OR, USA.
⁹ University of Göttingen, Institute for Psychology, Göttingen, Germany.
¹⁰ University of Southern California, Los Angeles, CA, USA.
¹¹ Australian National University, Canberra, Australia.
¹² Technische Universität Braunschweig, Braunschweig, Germany.
¹³ Parmenides Stiftung, Munich, Germany.
¹⁴ Queen's University, Kingston, Ontario, Canada.
¹⁵ Stanford University, Stanford, CA, USA.
¹⁶ Keele University, Keele, Staffordshire, UK.
¹⁷ Boston College, Chestnut Hill, MA, USA.
¹⁸ Radboud University Nijmegen, Nijmegen, Netherlands.
¹⁹ University of Koblenz-Landau, Landau, Germany.
²⁰ Erasmus Medical Center, Rotterdam, Netherlands.
²¹ University of Amsterdam, Amsterdam, Netherlands.
²² Harvard University, Cambridge, MA, USA.
²³ Occidental College, Los Angeles, CA, USA.
²⁴ Willamette University, Salem, OR, USA.
²⁵ Arcadia University, Glenside, PA, USA.
²⁶ University of Potsdam, Potsdam, Germany.
²⁷ University of Bristol, Bristol, UK.
²⁸ Vrije Universiteit Amsterdam, Amsterdam, Netherlands.
²⁹ Karolinska Institutet, Stockholm University, Stockholm, Sweden.
³⁰ Center for Open Science, Charlottesville, VA, USA. University of Virginia, Charlottesville, VA, USA. nosek@virginia.edu.
³¹ Harvard Medical School, Boston, MA, USA.
³² Loyola University, Baltimore, MD, USA.
³³ University of California, Riverside, CA, USA.
³⁴ Wesleyan University, Middletown, CT, USA.
³⁵ Center for Open Science, Charlottesville, VA, USA. University of Virginia, Charlottesville, VA, USA.
³⁶ University of Konstanz, Konstanz, Germany.
³⁷ Yale University, New Haven, CT, USA.
³⁸ Coventry University, Coventry, UK.
³⁹ Tilburg University, Tilburg, Netherlands.
⁴⁰ Tilburg University, Tilburg, Netherlands. Utrecht University, Utrecht, Netherlands.
⁴¹ University of Leuven, Leuven, Belgium.
⁴² University of Padova, Padova, Italy.
⁴³ University of Vienna, Vienna, Austria.
⁴⁴ Adams State University, Alamosa, CO, USA.

PMID: 26941312
DOI: 10.1126/science.aad9163

Abstract

Gilbert et al. conclude that evidence from the Open Science Collaboration's Reproducibility Project: Psychology indicates high reproducibility, given the study methodology. Their very optimistic assessment is limited by statistical misconceptions and by causal inferences from selectively interpreted, correlational data. Using the Reproducibility Project: Psychology data, both optimistic and pessimistic conclusions about reproducibility are possible, and neither are yet warranted.

PubMed Disclaimer

Comment in

Failure to replicate?
Maxcey AM. Maxcey AM. Atten Percept Psychophys. 2016 Apr;78(3):721-2. doi: 10.3758/s13414-016-1095-6. Atten Percept Psychophys. 2016. PMID: 26988566 No abstract available.

Comment on

PSYCHOLOGY. Estimating the reproducibility of psychological science.
Open Science Collaboration. Open Science Collaboration. Science. 2015 Aug 28;349(6251):aac4716. doi: 10.1126/science.aac4716. Science. 2015. PMID: 26315443
Comment on "Estimating the reproducibility of psychological science".
Gilbert DT, King G, Pettigrew S, Wilson TD. Gilbert DT, et al. Science. 2016 Mar 4;351(6277):1037. doi: 10.1126/science.aad7243. Science. 2016. PMID: 26941311

Cited by

Does health informatics have a replication crisis?
Coiera E, Ammenwerth E, Georgiou A, Magrabi F. Coiera E, et al. J Am Med Inform Assoc. 2018 Aug 1;25(8):963-968. doi: 10.1093/jamia/ocy028. J Am Med Inform Assoc. 2018. PMID: 29669066 Free PMC article.
A Systems Approach to Understanding and Improving Research Integrity.
Gorman DM, Elkins AD, Lawley M. Gorman DM, et al. Sci Eng Ethics. 2019 Feb;25(1):211-229. doi: 10.1007/s11948-017-9986-z. Epub 2017 Oct 25. Sci Eng Ethics. 2019. PMID: 29071573
Replication and the Establishment of Scientific Truth.
Iso-Ahola SE. Iso-Ahola SE. Front Psychol. 2020 Sep 16;11:2183. doi: 10.3389/fpsyg.2020.02183. eCollection 2020. Front Psychol. 2020. PMID: 33041887 Free PMC article. Review.
Building theories of consistency and variability in children's language development: A large-scale data approach.
Tsui ASM, Marchman VA, Frank MC. Tsui ASM, et al. Adv Child Dev Behav. 2021;61:199-221. doi: 10.1016/bs.acdb.2021.04.003. Epub 2021 Jun 14. Adv Child Dev Behav. 2021. PMID: 34266565 Free PMC article. Review.
The Weak Spots in Contemporary Science (and How to Fix Them).
Wicherts JM. Wicherts JM. Animals (Basel). 2017 Nov 27;7(12):90. doi: 10.3390/ani7120090. Animals (Basel). 2017. PMID: 29186879 Free PMC article. Review.

See all "Cited by" articles

Publication types

Actions
Actions

MeSH terms

Actions
Actions
Actions
Actions

LinkOut - more resources

Full Text Sources
Other Literature Sources
- scite Smart Citations

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Response to Comment on "Estimating the reproducibility of psychological science"

Affiliations

Response to Comment on "Estimating the reproducibility of psychological science"

Authors

Affiliations

Abstract

Comment in

Comment on

Similar articles

Cited by

Publication types

MeSH terms

LinkOut - more resources

Full Text Sources

Other Literature Sources