Variability in the analysis of a single neuroimaging dataset by many teams

Rotem Botvinik-Nezer^{1

2

3}, Felix Holzmeister⁴, Colin F Camerer⁵, Anna Dreber^{6

7}, Juergen Huber⁴, Magnus Johannesson⁶, Michael Kirchler⁴, Roni Iwanir^{1

2}, Jeanette A Mumford⁸, R Alison Adcock^{9

10}, Paolo Avesani^{11

12}, Blazej M Baczkowski¹³, Aahana Bajracharya¹⁴, Leah Bakst^{15

16}, Sheryl Ball^{17

18}, Marco Barilari¹⁹, Nadège Bault²⁰, Derek Beaton²¹, Julia Beitner^{22

23}, Roland G Benoit²⁴, Ruud M W J Berkers²⁴, Jamil P Bhanji²⁵, Bharat B Biswal^{26

27}, Sebastian Bobadilla-Suarez²⁸, Tiago Bortolini²⁹, Katherine L Bottenhorn³⁰, Alexander Bowring³¹, Senne Braem^{32

33}, Hayley R Brooks³⁴, Emily G Brudner²⁵, Cristian B Calderon³², Julia A Camilleri^{35

36}, Jaime J Castrellon^{9

37}, Luca Cecchetti³⁸, Edna C Cieslik^{35

36}, Zachary J Cole³⁹, Olivier Collignon^{12

19}, Robert W Cox⁴⁰, William A Cunningham⁴¹, Stefan Czoschke⁴², Kamalaker Dadi⁴³, Charles P Davis^{44

45

46}, Alberto De Luca⁴⁷, Mauricio R Delgado²⁵, Lysia Demetriou^{48

49}, Jeffrey B Dennison⁵⁰, Xin Di^{26

27}, Erin W Dickie^{51

52}, Ekaterina Dobryakova⁵³, Claire L Donnat⁵⁴, Juergen Dukart^{35

36}, Niall W Duncan^{55

56}, Joke Durnez⁵⁷, Amr Eed⁵⁸, Simon B Eickhoff^{35

36}, Andrew Erhart³⁴, Laura Fontanesi⁵⁹, G Matthew Fricke⁶⁰, Shiguang Fu^{61

62}, Adriana Galván⁶³, Remi Gau¹⁹, Sarah Genon^{35

36}, Tristan Glatard⁶⁴, Enrico Glerean⁶⁵, Jelle J Goeman⁶⁶, Sergej A E Golowin⁵⁵, Carlos González-García³², Krzysztof J Gorgolewski⁶⁷, Cheryl L Grady²¹, Mikella A Green^{9

37}, João F Guassi Moreira⁶³, Olivia Guest^{28

68}, Shabnam Hakimi⁹, J Paul Hamilton⁶⁹, Roeland Hancock^{45

46}, Giacomo Handjaras³⁸, Bronson B Harry⁷⁰, Colin Hawco⁷¹, Peer Herholz⁷², Gabrielle Herman⁷¹, Stephan Heunis^{73

74}, Felix Hoffstaedter^{35

36}, Jeremy Hogeveen^{75

76}, Susan Holmes⁵⁴, Chuan-Peng Hu⁷⁷, Scott A Huettel³⁷, Matthew E Hughes⁷⁸, Vittorio Iacovella¹², Alexandru D Iordan⁷⁹, Peder M Isager⁸⁰, Ayse I Isik⁸¹, Andrew Jahn⁸², Matthew R Johnson^{39

83}, Tom Johnstone⁷⁸, Michael J E Joseph⁷¹, Anthony C Juliano⁸⁴, Joseph W Kable^{85

86}, Michalis Kassinopoulos⁸⁷, Cemal Koba³⁸, Xiang-Zhen Kong⁸⁸, Timothy R Koscik⁸⁹, Nuri Erkut Kucukboyaci^{53

90}, Brice A Kuhl⁹¹, Sebastian Kupek⁹², Angela R Laird⁹³, Claus Lamm^{94

95}, Robert Langner^{35

36}, Nina Lauharatanahirun^{96

97}, Hongmi Lee⁹⁸, Sangil Lee⁸⁵, Alexander Leemans⁴⁷, Andrea Leo³⁸, Elise Lesage³², Flora Li^{99

100}, Monica Y C Li^{44

45

46

101}, Phui Cheng Lim^{39

83}, Evan N Lintz³⁹, Schuyler W Liphardt¹⁰², Annabel B Losecaat Vermeer⁹⁴, Bradley C Love^{28

103}, Michael L Mack⁴¹, Norberto Malpica¹⁰⁴, Theo Marins²⁹, Camille Maumet¹⁰⁵, Kelsey McDonald³⁷, Joseph T McGuire^{15

16}, Helena Melero^{104

106

107}, Adriana S Méndez Leal⁶³, Benjamin Meyer^{77

108}, Kristin N Meyer¹⁰⁹, Glad Mihai^{110

111}, Georgios D Mitsis¹¹², Jorge Moll^{29

67}, Dylan M Nielson¹¹³, Gustav Nilsonne^{114

115}, Michael P Notter¹¹⁶, Emanuele Olivetti^{11

12}, Adrian I Onicas³⁸, Paolo Papale^{38

117}, Kaustubh R Patil^{35

36}, Jonathan E Peelle¹⁴, Alexandre Pérez⁷², Doris Pischedda^{118

119

120}, Jean-Baptiste Poline^{72

121}, Yanina Prystauka^{44

45

46}, Shruti Ray²⁶, Patricia A Reuter-Lorenz⁷⁹, Richard C Reynolds¹²², Emiliano Ricciardi³⁸, Jenny R Rieck²¹, Anais M Rodriguez-Thompson¹⁰⁹, Anthony Romyn⁴¹, Taylor Salo³⁰, Gregory R Samanez-Larkin^{9

37}, Emilio Sanz-Morales¹⁰⁴, Margaret L Schlichting⁴¹, Douglas H Schultz^{39

83}, Qiang Shen^{61

62}, Margaret A Sheridan¹⁰⁹, Jennifer A Silvers⁶³, Kenny Skagerlund^{123

124}, Alec Smith^{17

18}, David V Smith⁵⁰, Peter Sokol-Hessner³⁴, Simon R Steinkamp¹²⁵, Sarah M Tashjian⁶³, Bertrand Thirion⁴³, John N Thorp¹²⁶, Gustav Tinghög^{127

128}, Loreen Tisdall^{67

129}, Steven H Tompson⁹⁶, Claudio Toro-Serey^{15

16}, Juan Jesus Torre Tresols⁴³, Leonardo Tozzi¹³⁰, Vuong Truong^{55

56}, Luca Turella¹², Anna E van 't Veer¹³¹, Tom Verguts³², Jean M Vettel^{132

133

134}, Sagana Vijayarajah⁴¹, Khoi Vo^{9

37}, Matthew B Wall^{135

136

137}, Wouter D Weeda¹³¹, Susanne Weis^{35

36}, David J White¹³⁸, David Wisniewski³², Alba Xifra-Porxas⁸⁷, Emily A Yearling^{44

45

46}, Sangsuk Yoon¹³⁹, Rui Yuan¹³⁰, Kenneth S L Yuen^{77

108}, Lei Zhang⁹⁴, Xu Zhang^{45

46

140}, Joshua E Zosky^{39

83}, Thomas E Nichols¹⁴¹, Russell A Poldrack¹⁴², Tom Schonberg^{143

144}

Affiliations

¹ Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel.
² Department of Neurobiology, The George S. Wise Faculty of Life Sciences, Tel Aviv University, Tel Aviv, Israel.
³ Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA.
⁴ Department of Banking and Finance, University of Innsbruck, Innsbruck, Austria.
⁵ HSS and CNS, California Institute of Technology, Pasadena, CA, USA.
⁶ Department of Economics, Stockholm School of Economics, Stockholm, Sweden.
⁷ Department of Economics, University of Innsbruck, Innsbruck, Austria.
⁸ Center for Healthy Minds, University of Wisconsin-Madison, Madison, WI, USA.
⁹ Center for Cognitive Neuroscience, Duke University, Durham, NC, USA.
¹⁰ Department of Psychiatry and Behavioral Sciences, Duke University, Durham, NC, USA.
¹¹ Neuroinformatics Laboratory, Fondazione Bruno Kessler, Trento, Italy.
¹² Center for Mind/Brain Sciences - CIMeC, University of Trento, Rovereto, Italy.
¹³ Department of Neurology, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.
¹⁴ Department of Otolaryngology, Washington University in St. Louis, St. Louis, MO, USA.
¹⁵ Department of Psychological and Brain Sciences, Boston University, Boston, MA, USA.
¹⁶ Center for Systems Neuroscience, Boston University, Boston, MA, USA.
¹⁷ Department of Economics, Virginia Tech, Blacksburg, VA, USA.
¹⁸ School of Neuroscience, Virginia Tech, Blacksburg, VA, USA.
¹⁹ Crossmodal Perception and Plasticity Laboratory, Institutes for Research in Psychology (IPSY) and Neurosciences (IoNS), UCLouvain, Louvain-la-Neuve, Belgium.
²⁰ School of Psychology, University of Plymouth, Plymouth, UK.
²¹ Rotman Research Institute, Baycrest Health Sciences Centre, Toronto, Ontario, Canada.
²² Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands.
²³ Department of Psychology, Goethe University, Frankfurt am Main, Germany.
²⁴ Max Planck Research Group: Adaptive Memory, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.
²⁵ Department of Psychology, Rutgers University-Newark, Newark, NJ, USA.
²⁶ Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ, USA.
²⁷ School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China.
²⁸ Department of Experimental Psychology, University College London, London, UK.
²⁹ D'Or Institute for Research and Education (IDOR), Rio de Janeiro, Brazil.
³⁰ Department of Psychology, Florida International University, Miami, FL, USA.
³¹ Oxford Big Data Institute, Li Ka Shing Centre for Health Information and Discovery, Nuffield Department of Population Health, University of Oxford, Oxford, UK.
³² Department of Experimental Psychology, Ghent University, Ghent, Belgium.
³³ Department of Psychology, Vrije Universiteit Brussel, Brussels, Belgium.
³⁴ Department of Psychology, University of Denver, Denver, CO, USA.
³⁵ Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Research Centre Jülich, Jülich, Germany.
³⁶ Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany.
³⁷ Department of Psychology and Neuroscience, Duke University, Durham, NC, USA.
³⁸ MoMiLab Research Unit, IMT School for Advanced Studies Lucca, Lucca, Italy.
³⁹ Department of Psychology, University of Nebraska-Lincoln, Lincoln, NE, USA.
⁴⁰ National Institute of Mental Health (NIMH), National Institutes of Health, Bethesda, MD, USA.
⁴¹ Department of Psychology, University of Toronto, Toronto, Ontario, Canada.
⁴² Institute of Medical Psychology, Goethe University, Frankfurt am Main, Germany.
⁴³ Inria, CEA, Université Paris-Saclay, Palaiseau, France.
⁴⁴ Department of Psychological Sciences, University of Connecticut, Storrs, CT, USA.
⁴⁵ Brain Imaging Research Center, University of Connecticut, Storrs, CT, USA.
⁴⁶ Connecticut Institute for the Brain and Cognitive Sciences, University of Connecticut, Storrs, CT, USA.
⁴⁷ PROVIDI Lab, Image Sciences Institute, University Medical Center Utrecht, Utrecht, The Netherlands.
⁴⁸ Section of Endocrinology and Investigative Medicine, Faculty of Medicine, Imperial College London, London, UK.
⁴⁹ Nuffield Department of Women's and Reproductive Health, University of Oxford, Oxford, UK.
⁵⁰ Department of Psychology, Temple University, Philadelphia, PA, USA.
⁵¹ Krembil Centre for Neuroinformatics, Campbell Family Mental Health Research Institute, Centre for Addiction and Mental Health, Toronto, Ontario, Canada.
⁵² Department of Psychiatry, University of Toronto, Toronto, Ontario, Canada.
⁵³ Center for Traumatic Brain Injury Research, Kessler Foundation, East Hanover, NJ, USA.
⁵⁴ Department of Statistics, Stanford University, Stanford, CA, USA.
⁵⁵ Graduate Institute of Mind, Brain and Consciousness, Taipei Medical University, Taipei, Taiwan.
⁵⁶ Brain and Consciousness Research Centre, TMU-ShuangHo Hospital, New Taipei City, Taiwan.
⁵⁷ Department of Psychology and Stanford Center for Reproducible Neuroscience, Stanford University, Stanford, CA, USA.
⁵⁸ Instituto de Neurociencias, CSIC-UMH, Alicante, Spain.
⁵⁹ Faculty of Psychology, University of Basel, Basel, Switzerland.
⁶⁰ Computer Science Department, University of New Mexico, Albuquerque, NM, USA.
⁶¹ School of Management, Zhejiang University of Technology, Hangzhou, China.
⁶² Institute of Neuromanagement, Zhejiang University of Technology, Hangzhou, China.
⁶³ Department of Psychology, University of California Los Angeles, Los Angeles, CA, USA.
⁶⁴ Department of Computer Science and Software Engineering, Concordia University, Montreal, Quebec, Canada.
⁶⁵ Department of Neuroscience and Biomedical Engineering, Aalto University, Espoo, Finland.
⁶⁶ Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands.
⁶⁷ Department of Psychology, Stanford University, Stanford, CA, USA.
⁶⁸ Research Centre on Interactive Media, Smart Systems and Emerging Technologies - RISE, Nicosia, Cyprus.
⁶⁹ Center for Social and Affective Neuroscience, Department of Biomedical and Clinical Sciences, Linköping University, Linköping, Sweden.
⁷⁰ The MARCS Institute for Brain, Behaviour and Development, Western Sydney University, Sydney, New South Wales, Australia.
⁷¹ Campbell Family Mental Health Research Institute, Centre for Addiction and Mental Health, Toronto, Ontario, Canada.
⁷² McConnell Brain Imaging Centre, The Neuro (Montreal Neurological Institute-Hospital), Faculty of Medicine, McGill University, Montreal, Quebec, Canada.
⁷³ Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands.
⁷⁴ Department of Research and Development, Epilepsy Centre Kempenhaeghe, Heeze, The Netherlands.
⁷⁵ Department of Psychology, University of New Mexico, Albuquerque, NM, USA.
⁷⁶ Psychology Clinical Neuroscience Center, University of New Mexico, Albuquerque, NM, USA.
⁷⁷ Leibniz-Institut für Resilienzforschung (LIR), Mainz, Germany.
⁷⁸ School of Health Sciences, Swinburne University of Technology, Hawthorn, Victoria, Australia.
⁷⁹ Department of Psychology, University of Michigan, Ann Arbor, MI, USA.
⁸⁰ Department of Industrial Engineering and Innovation Sciences, Eindhoven University of Technology, Eindhoven, The Netherlands.
⁸¹ Department of Neuroscience, Max Planck Institute for Empirical Aesthetics, Frankfurt am Main, Germany.
⁸² fMRI Laboratory, University of Michigan, Ann Arbor, MI, USA.
⁸³ Center for Brain, Biology and Behavior, University of Nebraska-Lincoln, Lincoln, NE, USA.
⁸⁴ Center for Neuropsychology and Neuroscience Research, Kessler Foundation, East Hanover, NJ, USA.
⁸⁵ Department of Psychology, University of Pennsylvania, Philadelphia, PA, USA.
⁸⁶ MindCORE, University of Pennsylvania, Philadelphia, PA, USA.
⁸⁷ Graduate Program in Biological and Biomedical Engineering, McGill University, Montreal, Quebec, Canada.
⁸⁸ Language and Genetics Department, Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands.
⁸⁹ Department of Psychiatry, University of Iowa Carver College of Medicine, Iowa City, IA, USA.
⁹⁰ Department of Physical Medicine and Rehabilitation, Rutgers New Jersey Medical School, Newark, NJ, USA.
⁹¹ Department of Psychology, University of Oregon, Eugene, OR, USA.
⁹² Faculty of Economics and Statistics, University of Innsbruck, Innsbruck, Austria.
⁹³ Department of Physics, Florida International University, Miami, Florida, USA.
⁹⁴ Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna, Austria.
⁹⁵ Vienna Cognitive Science Hub, University of Vienna, Vienna, Austria.
⁹⁶ US CCDC Army Research Laboratory, Human Research and Engineering Directorate, Aberdeen Proving Ground, MD, USA.
⁹⁷ Annenberg School for Communication, University of Pennsylvania, Philadelphia, PA, USA.
⁹⁸ Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD, USA.
⁹⁹ Fralin Biomedical Research Institute, Roanoke, VA, USA.
¹⁰⁰ Economics Experimental Lab, Nanjing Audit University, Nanjing, China.
¹⁰¹ Haskins Laboratories, New Haven, CT, USA.
¹⁰² Biology Department, University of New Mexico, Albuquerque, NM, USA.
¹⁰³ The Alan Turing Institute, London, UK.
¹⁰⁴ Laboratorio de Análisis de Imagen Médica y Biometría (LAIMBIO), Universidad Rey Juan Carlos, Madrid, Spain.
¹⁰⁵ Inria, Univ Rennes, CNRS, Inserm, IRISA UMR 6074, Empenn ERL U 1228, Rennes, France.
¹⁰⁶ Departamento de Psicobiología, División de Psicología, CES Cardenal Cisneros, Madrid, Spain.
¹⁰⁷ Northeastern University Biomedical Imaging Center, Northeastern University, Boston, MA, USA.
¹⁰⁸ Neuroimaging Center (NIC), Focus Program Translational Neurosciences (FTN), Johannes Gutenberg University Medical Center Mainz, Mainz, Germany.
¹⁰⁹ Department of Psychology and Neuroscience, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA.
¹¹⁰ Max Planck Research Group: Neural Mechanisms of Human Communication, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.
¹¹¹ Chair of Cognitive and Clinical Neuroscience, Faculty of Psychology, Technische Universität Dresden, Dresden, Germany.
¹¹² Department of Bioengineering, McGill University, Montreal, Quebec, Canada.
¹¹³ Data Science and Sharing Team, National Institute of Mental Health, National Institutes of Health, Bethesda, MD, USA.
¹¹⁴ Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden.
¹¹⁵ Department of Psychology, Stockholm University, Stockholm, Sweden.
¹¹⁶ The Laboratory for Investigative Neurophysiology (The LINE), Department of Radiology, University Hospital Center and University of Lausanne, Lausanne, Switzerland.
¹¹⁷ Department of Vision and Cognition, Netherlands Institute for Neuroscience, Amsterdam, The Netherlands.
¹¹⁸ Bernstein Center for Computational Neuroscience and Berlin Center for Advanced Neuroimaging and Clinic for Neurology, Charité Universitätsmedizin, corporate member of Freie Universität Berlin, Humboldt Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany.
¹¹⁹ Cluster of Excellence Science of Intelligence, Technische Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany.
¹²⁰ NeuroMI - Milan Center for Neuroscience, Milan, Italy.
¹²¹ Henry H. Wheeler, Jr. Brain Imaging Center, Helen Wills Neuroscience Institute, University of California Berkeley, Berkeley, CA, USA.
¹²² Scientific and Statistical Computing Core, National Institute of Mental Health, National Institutes of Health, Bethesda, MD, USA.
¹²³ Department of Behavioural Sciences and Learning, Linköping University, Linköping, Sweden.
¹²⁴ Center for Social and Affective Neuroscience, Department of Clinical and Experimental Medicine, Linköping University, Linköping, Sweden.
¹²⁵ Institute of Neuroscience and Medicine, Cognitive Neuroscience (INM-3), Research Centre Jülich, Jülich, Germany.
¹²⁶ Department of Psychology, Columbia University, New York, NY, USA.
¹²⁷ Department of Management and Engineering, Linköping University, Linköping, Sweden.
¹²⁸ Department of Health, Medicine and Caring Sciences, Linköping University, Linköping, Sweden.
¹²⁹ Center for Cognitive and Decision Sciences, University of Basel, Basel, Switzerland.
¹³⁰ Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA.
¹³¹ Methodology and Statistics Unit, Institute of Psychology, Leiden University, Leiden, The Netherlands.
¹³² US Combat Capabilities Development Command Army Research Laboratory, Aberdeen, MD, USA.
¹³³ University of California Santa Barbara, Santa Barbara, CA, USA.
¹³⁴ University of Pennsylvania, Philadelphia, PA, USA.
¹³⁵ Invicro, London, UK.
¹³⁶ Faculty of Medicine, Imperial College London, London, UK.
¹³⁷ Clinical Psychopharmacology Unit, University College London, London, UK.
¹³⁸ Centre for Human Psychopharmacology, Swinburne University, Hawthorn, Victoria, Australia.
¹³⁹ Department of Management and Marketing, School of Business, University of Dayton, Dayton, OH, USA.
¹⁴⁰ Biomedical Engineering Department, University of Connecticut, Storrs, CT, USA.
¹⁴¹ Oxford Big Data Institute, Li Ka Shing Centre for Health Information and Discovery, Nuffield Department of Population Health, University of Oxford, Oxford, UK. thomas.nichols@bdi.ox.ac.uk.
¹⁴² Department of Psychology, Stanford University, Stanford, CA, USA. poldrack@stanford.edu.
¹⁴³ Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel. schonberg@tauex.tau.ac.il.
¹⁴⁴ Department of Neurobiology, The George S. Wise Faculty of Life Sciences, Tel Aviv University, Tel Aviv, Israel. schonberg@tauex.tau.ac.il.

PMID: 32483374
PMCID: PMC7771346
DOI: 10.1038/s41586-020-2314-9

Variability in the analysis of a single neuroimaging dataset by many teams

Rotem Botvinik-Nezer et al. Nature. 2020 Jun.

. 2020 Jun;582(7810):84-88.

doi: 10.1038/s41586-020-2314-9. Epub 2020 May 20.

Authors

Affiliations

¹ Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel.
² Department of Neurobiology, The George S. Wise Faculty of Life Sciences, Tel Aviv University, Tel Aviv, Israel.
³ Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA.
⁴ Department of Banking and Finance, University of Innsbruck, Innsbruck, Austria.
⁵ HSS and CNS, California Institute of Technology, Pasadena, CA, USA.
⁶ Department of Economics, Stockholm School of Economics, Stockholm, Sweden.
⁷ Department of Economics, University of Innsbruck, Innsbruck, Austria.
⁸ Center for Healthy Minds, University of Wisconsin-Madison, Madison, WI, USA.
⁹ Center for Cognitive Neuroscience, Duke University, Durham, NC, USA.
¹⁰ Department of Psychiatry and Behavioral Sciences, Duke University, Durham, NC, USA.
¹¹ Neuroinformatics Laboratory, Fondazione Bruno Kessler, Trento, Italy.
¹² Center for Mind/Brain Sciences - CIMeC, University of Trento, Rovereto, Italy.
¹³ Department of Neurology, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.
¹⁴ Department of Otolaryngology, Washington University in St. Louis, St. Louis, MO, USA.
¹⁵ Department of Psychological and Brain Sciences, Boston University, Boston, MA, USA.
¹⁶ Center for Systems Neuroscience, Boston University, Boston, MA, USA.
¹⁷ Department of Economics, Virginia Tech, Blacksburg, VA, USA.
¹⁸ School of Neuroscience, Virginia Tech, Blacksburg, VA, USA.
¹⁹ Crossmodal Perception and Plasticity Laboratory, Institutes for Research in Psychology (IPSY) and Neurosciences (IoNS), UCLouvain, Louvain-la-Neuve, Belgium.
²⁰ School of Psychology, University of Plymouth, Plymouth, UK.
²¹ Rotman Research Institute, Baycrest Health Sciences Centre, Toronto, Ontario, Canada.
²² Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands.
²³ Department of Psychology, Goethe University, Frankfurt am Main, Germany.
²⁴ Max Planck Research Group: Adaptive Memory, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.
²⁵ Department of Psychology, Rutgers University-Newark, Newark, NJ, USA.
²⁶ Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ, USA.
²⁷ School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China.
²⁸ Department of Experimental Psychology, University College London, London, UK.
²⁹ D'Or Institute for Research and Education (IDOR), Rio de Janeiro, Brazil.
³⁰ Department of Psychology, Florida International University, Miami, FL, USA.
³¹ Oxford Big Data Institute, Li Ka Shing Centre for Health Information and Discovery, Nuffield Department of Population Health, University of Oxford, Oxford, UK.
³² Department of Experimental Psychology, Ghent University, Ghent, Belgium.
³³ Department of Psychology, Vrije Universiteit Brussel, Brussels, Belgium.
³⁴ Department of Psychology, University of Denver, Denver, CO, USA.
³⁵ Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Research Centre Jülich, Jülich, Germany.
³⁶ Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany.
³⁷ Department of Psychology and Neuroscience, Duke University, Durham, NC, USA.
³⁸ MoMiLab Research Unit, IMT School for Advanced Studies Lucca, Lucca, Italy.
³⁹ Department of Psychology, University of Nebraska-Lincoln, Lincoln, NE, USA.
⁴⁰ National Institute of Mental Health (NIMH), National Institutes of Health, Bethesda, MD, USA.
⁴¹ Department of Psychology, University of Toronto, Toronto, Ontario, Canada.
⁴² Institute of Medical Psychology, Goethe University, Frankfurt am Main, Germany.
⁴³ Inria, CEA, Université Paris-Saclay, Palaiseau, France.
⁴⁴ Department of Psychological Sciences, University of Connecticut, Storrs, CT, USA.
⁴⁵ Brain Imaging Research Center, University of Connecticut, Storrs, CT, USA.
⁴⁶ Connecticut Institute for the Brain and Cognitive Sciences, University of Connecticut, Storrs, CT, USA.
⁴⁷ PROVIDI Lab, Image Sciences Institute, University Medical Center Utrecht, Utrecht, The Netherlands.
⁴⁸ Section of Endocrinology and Investigative Medicine, Faculty of Medicine, Imperial College London, London, UK.
⁴⁹ Nuffield Department of Women's and Reproductive Health, University of Oxford, Oxford, UK.
⁵⁰ Department of Psychology, Temple University, Philadelphia, PA, USA.
⁵¹ Krembil Centre for Neuroinformatics, Campbell Family Mental Health Research Institute, Centre for Addiction and Mental Health, Toronto, Ontario, Canada.
⁵² Department of Psychiatry, University of Toronto, Toronto, Ontario, Canada.
⁵³ Center for Traumatic Brain Injury Research, Kessler Foundation, East Hanover, NJ, USA.
⁵⁴ Department of Statistics, Stanford University, Stanford, CA, USA.
⁵⁵ Graduate Institute of Mind, Brain and Consciousness, Taipei Medical University, Taipei, Taiwan.
⁵⁶ Brain and Consciousness Research Centre, TMU-ShuangHo Hospital, New Taipei City, Taiwan.
⁵⁷ Department of Psychology and Stanford Center for Reproducible Neuroscience, Stanford University, Stanford, CA, USA.
⁵⁸ Instituto de Neurociencias, CSIC-UMH, Alicante, Spain.
⁵⁹ Faculty of Psychology, University of Basel, Basel, Switzerland.
⁶⁰ Computer Science Department, University of New Mexico, Albuquerque, NM, USA.
⁶¹ School of Management, Zhejiang University of Technology, Hangzhou, China.
⁶² Institute of Neuromanagement, Zhejiang University of Technology, Hangzhou, China.
⁶³ Department of Psychology, University of California Los Angeles, Los Angeles, CA, USA.
⁶⁴ Department of Computer Science and Software Engineering, Concordia University, Montreal, Quebec, Canada.
⁶⁵ Department of Neuroscience and Biomedical Engineering, Aalto University, Espoo, Finland.
⁶⁶ Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands.
⁶⁷ Department of Psychology, Stanford University, Stanford, CA, USA.
⁶⁸ Research Centre on Interactive Media, Smart Systems and Emerging Technologies - RISE, Nicosia, Cyprus.
⁶⁹ Center for Social and Affective Neuroscience, Department of Biomedical and Clinical Sciences, Linköping University, Linköping, Sweden.
⁷⁰ The MARCS Institute for Brain, Behaviour and Development, Western Sydney University, Sydney, New South Wales, Australia.
⁷¹ Campbell Family Mental Health Research Institute, Centre for Addiction and Mental Health, Toronto, Ontario, Canada.
⁷² McConnell Brain Imaging Centre, The Neuro (Montreal Neurological Institute-Hospital), Faculty of Medicine, McGill University, Montreal, Quebec, Canada.
⁷³ Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands.
⁷⁴ Department of Research and Development, Epilepsy Centre Kempenhaeghe, Heeze, The Netherlands.
⁷⁵ Department of Psychology, University of New Mexico, Albuquerque, NM, USA.
⁷⁶ Psychology Clinical Neuroscience Center, University of New Mexico, Albuquerque, NM, USA.
⁷⁷ Leibniz-Institut für Resilienzforschung (LIR), Mainz, Germany.
⁷⁸ School of Health Sciences, Swinburne University of Technology, Hawthorn, Victoria, Australia.
⁷⁹ Department of Psychology, University of Michigan, Ann Arbor, MI, USA.
⁸⁰ Department of Industrial Engineering and Innovation Sciences, Eindhoven University of Technology, Eindhoven, The Netherlands.
⁸¹ Department of Neuroscience, Max Planck Institute for Empirical Aesthetics, Frankfurt am Main, Germany.
⁸² fMRI Laboratory, University of Michigan, Ann Arbor, MI, USA.
⁸³ Center for Brain, Biology and Behavior, University of Nebraska-Lincoln, Lincoln, NE, USA.
⁸⁴ Center for Neuropsychology and Neuroscience Research, Kessler Foundation, East Hanover, NJ, USA.
⁸⁵ Department of Psychology, University of Pennsylvania, Philadelphia, PA, USA.
⁸⁶ MindCORE, University of Pennsylvania, Philadelphia, PA, USA.
⁸⁷ Graduate Program in Biological and Biomedical Engineering, McGill University, Montreal, Quebec, Canada.
⁸⁸ Language and Genetics Department, Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands.
⁸⁹ Department of Psychiatry, University of Iowa Carver College of Medicine, Iowa City, IA, USA.
⁹⁰ Department of Physical Medicine and Rehabilitation, Rutgers New Jersey Medical School, Newark, NJ, USA.
⁹¹ Department of Psychology, University of Oregon, Eugene, OR, USA.
⁹² Faculty of Economics and Statistics, University of Innsbruck, Innsbruck, Austria.
⁹³ Department of Physics, Florida International University, Miami, Florida, USA.
⁹⁴ Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna, Austria.
⁹⁵ Vienna Cognitive Science Hub, University of Vienna, Vienna, Austria.
⁹⁶ US CCDC Army Research Laboratory, Human Research and Engineering Directorate, Aberdeen Proving Ground, MD, USA.
⁹⁷ Annenberg School for Communication, University of Pennsylvania, Philadelphia, PA, USA.
⁹⁸ Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD, USA.
⁹⁹ Fralin Biomedical Research Institute, Roanoke, VA, USA.
¹⁰⁰ Economics Experimental Lab, Nanjing Audit University, Nanjing, China.
¹⁰¹ Haskins Laboratories, New Haven, CT, USA.
¹⁰² Biology Department, University of New Mexico, Albuquerque, NM, USA.
¹⁰³ The Alan Turing Institute, London, UK.
¹⁰⁴ Laboratorio de Análisis de Imagen Médica y Biometría (LAIMBIO), Universidad Rey Juan Carlos, Madrid, Spain.
¹⁰⁵ Inria, Univ Rennes, CNRS, Inserm, IRISA UMR 6074, Empenn ERL U 1228, Rennes, France.
¹⁰⁶ Departamento de Psicobiología, División de Psicología, CES Cardenal Cisneros, Madrid, Spain.
¹⁰⁷ Northeastern University Biomedical Imaging Center, Northeastern University, Boston, MA, USA.
¹⁰⁸ Neuroimaging Center (NIC), Focus Program Translational Neurosciences (FTN), Johannes Gutenberg University Medical Center Mainz, Mainz, Germany.
¹⁰⁹ Department of Psychology and Neuroscience, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA.
¹¹⁰ Max Planck Research Group: Neural Mechanisms of Human Communication, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.
¹¹¹ Chair of Cognitive and Clinical Neuroscience, Faculty of Psychology, Technische Universität Dresden, Dresden, Germany.
¹¹² Department of Bioengineering, McGill University, Montreal, Quebec, Canada.
¹¹³ Data Science and Sharing Team, National Institute of Mental Health, National Institutes of Health, Bethesda, MD, USA.
¹¹⁴ Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden.
¹¹⁵ Department of Psychology, Stockholm University, Stockholm, Sweden.
¹¹⁶ The Laboratory for Investigative Neurophysiology (The LINE), Department of Radiology, University Hospital Center and University of Lausanne, Lausanne, Switzerland.
¹¹⁷ Department of Vision and Cognition, Netherlands Institute for Neuroscience, Amsterdam, The Netherlands.
¹¹⁸ Bernstein Center for Computational Neuroscience and Berlin Center for Advanced Neuroimaging and Clinic for Neurology, Charité Universitätsmedizin, corporate member of Freie Universität Berlin, Humboldt Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany.
¹¹⁹ Cluster of Excellence Science of Intelligence, Technische Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany.
¹²⁰ NeuroMI - Milan Center for Neuroscience, Milan, Italy.
¹²¹ Henry H. Wheeler, Jr. Brain Imaging Center, Helen Wills Neuroscience Institute, University of California Berkeley, Berkeley, CA, USA.
¹²² Scientific and Statistical Computing Core, National Institute of Mental Health, National Institutes of Health, Bethesda, MD, USA.
¹²³ Department of Behavioural Sciences and Learning, Linköping University, Linköping, Sweden.
¹²⁴ Center for Social and Affective Neuroscience, Department of Clinical and Experimental Medicine, Linköping University, Linköping, Sweden.
¹²⁵ Institute of Neuroscience and Medicine, Cognitive Neuroscience (INM-3), Research Centre Jülich, Jülich, Germany.
¹²⁶ Department of Psychology, Columbia University, New York, NY, USA.
¹²⁷ Department of Management and Engineering, Linköping University, Linköping, Sweden.
¹²⁸ Department of Health, Medicine and Caring Sciences, Linköping University, Linköping, Sweden.
¹²⁹ Center for Cognitive and Decision Sciences, University of Basel, Basel, Switzerland.
¹³⁰ Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA.
¹³¹ Methodology and Statistics Unit, Institute of Psychology, Leiden University, Leiden, The Netherlands.
¹³² US Combat Capabilities Development Command Army Research Laboratory, Aberdeen, MD, USA.
¹³³ University of California Santa Barbara, Santa Barbara, CA, USA.
¹³⁴ University of Pennsylvania, Philadelphia, PA, USA.
¹³⁵ Invicro, London, UK.
¹³⁶ Faculty of Medicine, Imperial College London, London, UK.
¹³⁷ Clinical Psychopharmacology Unit, University College London, London, UK.
¹³⁸ Centre for Human Psychopharmacology, Swinburne University, Hawthorn, Victoria, Australia.
¹³⁹ Department of Management and Marketing, School of Business, University of Dayton, Dayton, OH, USA.
¹⁴⁰ Biomedical Engineering Department, University of Connecticut, Storrs, CT, USA.
¹⁴¹ Oxford Big Data Institute, Li Ka Shing Centre for Health Information and Discovery, Nuffield Department of Population Health, University of Oxford, Oxford, UK. thomas.nichols@bdi.ox.ac.uk.
¹⁴² Department of Psychology, Stanford University, Stanford, CA, USA. poldrack@stanford.edu.
¹⁴³ Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel. schonberg@tauex.tau.ac.il.
¹⁴⁴ Department of Neurobiology, The George S. Wise Faculty of Life Sciences, Tel Aviv University, Tel Aviv, Israel. schonberg@tauex.tau.ac.il.

PMID: 32483374
PMCID: PMC7771346
DOI: 10.1038/s41586-020-2314-9

Abstract

Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses¹. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset^2-5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.

PubMed Disclaimer

Figures

**Extended Data Figure 1 ∣. Voxels overlap.**
Maps showing at each voxel the proportion of teams (out of N = 65 teams) reporting significant activations in their thresholded statistical map, for each hypothesis (labeled H1 - H9), thresholded at 10% (i.e., voxels with no color were significant in fewer than 10% of teams). +/− refers to direction of effect, gain/loss refers to the effect being tested, and equal indifference (EI) / equal range (ER) refers to the group being examined or compared. Hypotheses #1 and #3, as well as hypotheses #2 and #4, share the same statistical maps as the hypotheses are for the same contrast and experimental group, but for different regions (see Extended Data Table 1). Images can be viewed at https://identifiers.org/neurovault.collection:6047

**Extended Data Figure 2 ∣. Variability of whole-brain unthresholded maps for hypotheses 2, 4 - 9.**
For each hypothesis, we present a heatmap based on Spearman correlations between unthresholded statistical maps (N = 64), clustered according to their similarity, and the average of unthresholded images for each cluster (cluster colors in titles refer to colors in left margin of heatmap). Green / red at the columns represent binary results (significant / not significant, respectively) reported by the analysis teams; row colors represent cluster membership. Maps are thresholded at an uncorrected value of Z > 2 for visualization. Unthresholded maps for Hypothesis #2 and Hypothesis #4 are identical (as they both relate to the same contrast and group, but different regions), and the colors represent reported results for Hypothesis #2. For Hypotheses #1 and #3 see Figure 2.

**Extended Data Figure 3 ∣. Variability and consensus of unthresholded statistical maps (N = 64).**
(a) Maps of estimated between-team variability (tau) at each voxel for each hypothesis. Images can be viewed at https://identifiers.org/neurovault.collection:6050. (b) Image-based meta-analysis (IBMA) results. A consensus analysis was performed on the unthresholded statistical maps to obtain a group statistical map for each hypothesis, accounting for the correlation between teams due to the same underlying data (see Methods). Maps are presented for each hypothesis showing voxels (in color) where the group statistic was significantly greater than zero after voxelwise correction for false discovery rate (p < 0.05). Color bar reflects statistical value (Z) for the meta-analysis. Images can be viewed at https://identifiers.org/neurovault.collection:6051. Hypotheses #1 and # 3, as well as Hypotheses #2 and #4, share the same unthresholded maps, as they relate to the same contrast and group but for different regions (see Extended Data Table 1).

**Extended Data Figure 4 ∣. Results of the consistent thresholding and ROI selection analysis (N = 64).**
(a) Activation for each hypothesis as determined using consistent thresholding (black: p < 0.001 and *cluster size* > 10 voxels; blue: FDR correction with p < 0.05) and ROI selection across teams (y-axis), versus actual proportion of teams reporting activation (x-axis). Numbers next to each symbol represent the hypothesis number for each point. (b) Results from re-thresholding of unthresholded maps using uncorrected (p < 0.001, cluster size k > 10) and false discovery rate correction (*pFDR* < 5%) and common anatomical regions of interest for each hypothesis. A team is recorded as having an activation if one or more significant voxels are found in the ROI. Results for image-based meta-analysis (IBMA) for each hypothesis are also presented, thresholded at pFDR < 5% as well.

**Extended Data Figure 5 ∣. Prediction markets over time (N = 240 observations [10 days X 24 hours]).**
(a). Panel regressions. The table summarizes the results of pre-registered fixed-effects panel regressions of the predictions absolute errors (i.e., the absolute deviation of the market price from the fundamental value) on an hourly basis (average price of all transactions within an hour) on time and prediction market indicators. Standard errors are computed using a robust estimator. (b) Market prices for each of the nine hypotheses separated for the team members (green) and non-team members (blue) prediction markets. The figure shows the average prediction market prices per hour separated for the two prediction markets for the time the markets were open (10 days, i.e., 240 hours). The gray line indicates the actual share of analysis teams reporting a significant result for the hypothesis (i.e., the fundamental value).

**Figure 1:. Fraction of teams reporting a significant result and prediction market beliefs.**
The figure depicts final market prices for the “team members” (blue dots; N = 83 active traders) and the “non-team members” (green dots; N = 65 active traders) markets as well as the as well as the observed fraction of teams reporting significant results (fundamental value, pink dots; N = 70 analysis teams), and the corresponding 95% confidence intervals for each of the nine hypotheses (note that the hypotheses are sorted based on the fundamental value). Confidence intervals were constructed by assuming convergence of the binomial distribution towards the normal.

**Figure 2.. Analytic variability in whole-brain statistical results for Hypothesis 1.**
**Top panel**: Spearman correlation values between whole-brain unthresholded statistical maps for each team (N = 64) were computed and clustered according to their similarity (using Ward clustering on Euclidean distances). Row colors (left) denote cluster membership, while column colors (top) represent hypothesis decisions (green: Yes, red: No). Brackets represent clustering. **Bottom panel**: Average statistical maps (thresholded at uncorrected z > 2.0) for each of the three clusters depicted in the left panel. The probability of reporting a positive hypothesis outcome (pYes) is presented for each cluster. Images can be viewed at https://identifiers.org/neurovault.collection:6048.

See this image and copyright information in PMC

Comment in

Neuroimaging results altered by varying analysis pipelines.
Lindquist M. Lindquist M. Nature. 2020 Jun;582(7810):36-37. doi: 10.1038/d41586-020-01282-z. Nature. 2020. PMID: 32433631 No abstract available.
One statistical analysis must not rule them all.
Wagenmakers EJ, Sarafoglou A, Aczel B. Wagenmakers EJ, et al. Nature. 2022 May;605(7910):423-425. doi: 10.1038/d41586-022-01332-8. Nature. 2022. PMID: 35581494 No abstract available.

References

1. Botvinik-Nezer R et al. fMRI data of mixed gambles from the Neuroimaging Analysis Replication and Prediction Study. Scientific Data 6, 1–9 (2019). - PMC - PubMed
1. Dreber A et al. Using prediction markets to estimate the reproducibility of scientific research. Proceedings of the National Academy of Sciences 112, 15343–15347 (2015). - PMC - PubMed
1. Camerer CF et al. Evaluating replicability of laboratory experiments in economics. Science 351, 1433–1436 (2016). - PubMed
1. Camerer CF et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour 2, 637–644 (2018). - PubMed
1. Forsell E et al. Predicting replication outcomes in the Many Labs 2 study. J. Econ. Psychol 75, 102117 (2019).

Methods References

1. Gorgolewski KJ et al. The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments. Sci Data 3, 1–9 (2016). - PMC - PubMed
1. Tversky A & Kahneman D Advances in Prospect Theory: Cumulative Representation of Uncertainty. J. Risk Uncertain 5, 297–323 (1992).
1. Nichols TE et al. Best practices in data analysis and sharing in neuroimaging using MRI. Nat. Neurosci 20, 299–303 (2017). - PMC - PubMed
1. Bates D, Mächler M, Bolker B & Walker S Fitting Linear Mixed-Effects Models Using lme4. J. Stat. Softw 67, 1–48 (2015).
1. Lubke GH et al. Assessing Model Selection Uncertainty Using a Bootstrap Approach: An update. Struct. Equ. Modeling 24, 230–245 (2017). - PMC - PubMed

Publication types

Actions
Actions
Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Variability in the analysis of a single neuroimaging dataset by many teams

Affiliations

Variability in the analysis of a single neuroimaging dataset by many teams

Authors

Affiliations

Abstract

Figures

Comment in

References

Methods References

Publication types

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources

Medical