Sustainable computational science: the ReScience initiative

Nicolas P Rougier¹, Konrad Hinsen², Frédéric Alexandre¹, Thomas Arildsen³, Lorena A Barba⁴, Fabien C Y Benureau¹, C Titus Brown⁵, Pierre de Buyl⁶, Ozan Caglayan⁷, Andrew P Davison⁸, Marc-André Delsuc⁹, Georgios Detorakis¹⁰, Alexandra K Diem¹¹, Damien Drix¹², Pierre Enel¹³, Benoît Girard¹⁴, Olivia Guest¹⁵, Matt G Hall¹⁶, Rafael N Henriques¹⁷, Xavier Hinaut¹, Kamil S Jaron¹⁸, Mehdi Khamassi¹⁴, Almar Klein¹⁹, Tiina Manninen²⁰, Pietro Marchesi²¹, Daniel McGlinn²², Christoph Metzner²³, Owen Petchey²⁴, Hans Ekkehard Plesser²⁵, Timothée Poisot²⁶, Karthik Ram²⁷, Yoav Ram²⁸, Etienne Roesch²⁹, Cyrille Rossant³⁰, Vahid Rostami³¹, Aaron Shifman³², Jemma Stachelek³³, Marcel Stimberg³⁴, Frank Stollmeier³⁵, Federico Vaggi³⁶, Guillaume Viejo¹⁴, Julien Vitay³⁷, Anya E Vostinar³⁸, Roman Yurchak³⁹, Tiziano Zito⁴⁰

Affiliations

¹ INRIA Bordeaux Sud-Ouest, Talence, France.
² Centre de Biophysique Moléculaire UPR4301, CNRS, Orléans, France.
³ Department of Electronic Systems, Technical Faculty of IT and Design, Aalborg University, Aalborg, Denmark.
⁴ Department of Mechanical and Aerospace Engineering, The George Washington University, Washington, D.C., USA.
⁵ Department of Population Health and Reproduction, University of California Davis, Davis, CA, USA.
⁶ Instituut voor Theoretische Fysica, KU Leuven, Leuven, Belgium.
⁷ Laboratoire d'Informatique (LIUM), Le Mans University, Le Mans, France.
⁸ UNIC FRE 3693, CNRS, Gif-sur-Yvette, France.
⁹ Institut de Génétique et de Biologie Moléculaire et Cellulaire, Illkirch, France.
¹⁰ Department of Cognitive Sciences, University of California Irvine, Irvine, CA, USA.
¹¹ Computational Engineering and Design, University of Southampton, Southampton, United Kingdom.
¹² Humboldt Universität zu Berlin, Berlin, Germany.
¹³ Department of Neuroscience, Mount Sinai School of Medicine, New York, NY, USA.
¹⁴ Institute of Intelligent Systems and Robotics, Sorbonne Universités - UPMC Univ Paris 06 - CNRS, Paris, France.
¹⁵ Experimental Psychology, University College London, London, Greater London, United Kingdom.
¹⁶ UCL Great Ormond St Institute of Child Health, London, United Kingdom.
¹⁷ Champalimaud Centre for the Unknown, Champalimaud Neuroscience Program, Lisbon, Portugal.
¹⁸ Department of Ecology and Evolution, University of Lausanne, Lausanne, Switzerland.
¹⁹ Independent scholar, Enschede, The Netherlands.
²⁰ BioMediTech Institute and Faculty of Biomedical Sciences and Engineering, Tampere University of Technology, Tampere, Finland.
²¹ Swammerdam Institute for Life Sciences, University of Amsterdam, Amsterdam, The Netherlands.
²² Department of Biology, College of Charleston, Charleston, SC, USA.
²³ Centre for Computer Science and Informatics Research, University of Hertfordshire, Hatfield, United Kingdom.
²⁴ Department of Evolutionary Biology and Environmental Studies, University of Zurich, Zurich, Switzerland.
²⁵ Faculty of Science and Technology, Norwegian University of Life Sciences, Aas, Norway.
²⁶ Département de Sciences Biologiques, Université de Montréal, Montréal, QC, Canada.
²⁷ Berkeley Institute for Data Science, University of California, Berkeley, CA, USA.
²⁸ Department of Biology, Stanford University, Stanford, CA, USA.
²⁹ Centre for Integrative Neuroscience, University of Reading, Reading, United Kingdom.
³⁰ Institute of Neurology, University College London, London, United Kingdom.
³¹ Institute of Neuroscience & Medicine, Juelich Forschungszentrum, Jülich, Germany.
³² Department of Biology, University of Ottawa, Ottawa, Ontario, Canada.
³³ Department of Fisheries and Wildlife, Michigan State University, East Lansing, MI, USA.
³⁴ Sorbonne Universités/UPMC Univ Paris 06/INSERM/CNRS/Institut de la Vision, Paris, France.
³⁵ Max Planck Institute for Dynamics and Self-Organization, Göttingen, Lower Saxony, Germany.
³⁶ Amazon, Seattle, WA, USA.
³⁷ Department of Computer Science, Chemnitz University of Technology, Chemnitz, Saxony, Germany.
³⁸ Department of Computer Science, Grinnell College, Grinnell, IA, USA.
³⁹ Symerio, Palaiseau, France.
⁴⁰ Neural Information Processing Group, Eberhard Karls Universität Tübingen, Tübingen, Germany.

PMID: 34722870
PMCID: PMC8530091
DOI: 10.7717/peerj-cs.142

Sustainable computational science: the ReScience initiative

Nicolas P Rougier et al. PeerJ Comput Sci. 2017.

. 2017 Dec 18:3:e142.

doi: 10.7717/peerj-cs.142. eCollection 2017.

Authors

Affiliations

¹ INRIA Bordeaux Sud-Ouest, Talence, France.
² Centre de Biophysique Moléculaire UPR4301, CNRS, Orléans, France.
³ Department of Electronic Systems, Technical Faculty of IT and Design, Aalborg University, Aalborg, Denmark.
⁴ Department of Mechanical and Aerospace Engineering, The George Washington University, Washington, D.C., USA.
⁵ Department of Population Health and Reproduction, University of California Davis, Davis, CA, USA.
⁶ Instituut voor Theoretische Fysica, KU Leuven, Leuven, Belgium.
⁷ Laboratoire d'Informatique (LIUM), Le Mans University, Le Mans, France.
⁸ UNIC FRE 3693, CNRS, Gif-sur-Yvette, France.
⁹ Institut de Génétique et de Biologie Moléculaire et Cellulaire, Illkirch, France.
¹⁰ Department of Cognitive Sciences, University of California Irvine, Irvine, CA, USA.
¹¹ Computational Engineering and Design, University of Southampton, Southampton, United Kingdom.
¹² Humboldt Universität zu Berlin, Berlin, Germany.
¹³ Department of Neuroscience, Mount Sinai School of Medicine, New York, NY, USA.
¹⁴ Institute of Intelligent Systems and Robotics, Sorbonne Universités - UPMC Univ Paris 06 - CNRS, Paris, France.
¹⁵ Experimental Psychology, University College London, London, Greater London, United Kingdom.
¹⁶ UCL Great Ormond St Institute of Child Health, London, United Kingdom.
¹⁷ Champalimaud Centre for the Unknown, Champalimaud Neuroscience Program, Lisbon, Portugal.
¹⁸ Department of Ecology and Evolution, University of Lausanne, Lausanne, Switzerland.
¹⁹ Independent scholar, Enschede, The Netherlands.
²⁰ BioMediTech Institute and Faculty of Biomedical Sciences and Engineering, Tampere University of Technology, Tampere, Finland.
²¹ Swammerdam Institute for Life Sciences, University of Amsterdam, Amsterdam, The Netherlands.
²² Department of Biology, College of Charleston, Charleston, SC, USA.
²³ Centre for Computer Science and Informatics Research, University of Hertfordshire, Hatfield, United Kingdom.
²⁴ Department of Evolutionary Biology and Environmental Studies, University of Zurich, Zurich, Switzerland.
²⁵ Faculty of Science and Technology, Norwegian University of Life Sciences, Aas, Norway.
²⁶ Département de Sciences Biologiques, Université de Montréal, Montréal, QC, Canada.
²⁷ Berkeley Institute for Data Science, University of California, Berkeley, CA, USA.
²⁸ Department of Biology, Stanford University, Stanford, CA, USA.
²⁹ Centre for Integrative Neuroscience, University of Reading, Reading, United Kingdom.
³⁰ Institute of Neurology, University College London, London, United Kingdom.
³¹ Institute of Neuroscience & Medicine, Juelich Forschungszentrum, Jülich, Germany.
³² Department of Biology, University of Ottawa, Ottawa, Ontario, Canada.
³³ Department of Fisheries and Wildlife, Michigan State University, East Lansing, MI, USA.
³⁴ Sorbonne Universités/UPMC Univ Paris 06/INSERM/CNRS/Institut de la Vision, Paris, France.
³⁵ Max Planck Institute for Dynamics and Self-Organization, Göttingen, Lower Saxony, Germany.
³⁶ Amazon, Seattle, WA, USA.
³⁷ Department of Computer Science, Chemnitz University of Technology, Chemnitz, Saxony, Germany.
³⁸ Department of Computer Science, Grinnell College, Grinnell, IA, USA.
³⁹ Symerio, Palaiseau, France.
⁴⁰ Neural Information Processing Group, Eberhard Karls Universität Tübingen, Tübingen, Germany.

PMID: 34722870
PMCID: PMC8530091
DOI: 10.7717/peerj-cs.142

Abstract

Computer science offers a large set of tools for prototyping, writing, running, testing, validating, sharing and reproducing results; however, computational science lags behind. In the best case, authors may provide their source code as a compressed archive and they may feel confident their research is reproducible. But this is not exactly true. James Buckheit and David Donoho proposed more than two decades ago that an article about computational results is advertising, not scholarship. The actual scholarship is the full software environment, code, and data that produced the result. This implies new workflows, in particular in peer-reviews. Existing journals have been slow to adapt: source codes are rarely requested and are hardly ever actually executed to check that they produce the results advertised in the article. ReScience is a peer-reviewed journal that targets computational research and encourages the explicit replication of already published research, promoting new and open-source implementations in order to ensure that the original research can be replicated from its description. To achieve this goal, the whole publishing chain is radically different from other traditional scientific journals. ReScience resides on GitHub where each new implementation of a computational study is made available together with comments, explanations, and software tests.

Keywords: Computational science; GitHub; Open peer-review; Open science; Publication; Replicable; Reproducible; Sustainable.

PubMed Disclaimer

Conflict of interest statement

Federico Vaggi is an employee of Amazon, Inc., Roman Yurchak is an employee of Symerio, and C. Titus Brown and Nicolas P. Rougier are Academic Editors for PeerJ.

Figures

**Figure 1**
(A) The ReScience publication chain starts from an original research article by authors A, published in a journal, in conference proceedings, or as a preprint. This article constitutes the base material for authors B, who attempt to replicate the work based on its description. Success or failure to replicate is not a criterion for acceptance or rejection, even though failure to replicate requires more precaution to ensure this is not a misunderstanding or a bug in the new code. After review, the replication is published, and feedback is given to original authors (and editors) to inform them the work has been replicated (or not). (B) The CoScience proposal would require the replication to happen *before* the actual publication. In case of failure, nothing will be published. In case of success, the publication will be endorsed by authors A and authors B with identified roles and will be certified as reproducible because it has been replicated by an independent group.

See this image and copyright information in PMC

References

1. Anaconda Inc Conda. 2017. https://conda.io/ https://conda.io/
1. Baker M. 1, 500 scientists lift the lid on reproducibility. Nature. 2016;533(7604):452–454. doi: 10.1038/533452a. - DOI - PubMed
1. Colberg C, Proebsting TA. Repeatability in computer systems research. Communications of the ACM. 2016;59(3):62–69. doi: 10.1145/2812803. - DOI
1. Coudert F-X. Reproducible research in computational chemistry of materials. Chemistry of Materials. 2017;29(7):2615–2617. doi: 10.1021/acs.chemmater.7b00799. - DOI
1. Courtès L, Wurmus R. Reproducible and user-controlled software environments in HPC with Guix. In: Hunold S, Costan A, Giménez D, Iosup A, Ricci L, Requena MEG, Scarano V, Varbanescu AL, Scott SL, Lankes S, Weidendorfer J, Alexander M, editors. Euro-Par 2015: parallel processing workshops. vol. 9523. Springer; Cham: 2015. (Lecture notes in computer science).

LinkOut - more resources

Full Text Sources

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Sustainable computational science: the ReScience initiative

Affiliations

Sustainable computational science: the ReScience initiative

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

References

LinkOut - more resources

Full Text Sources