Reproducibility in machine learning for health research: Still a ways to go

Matthew B A McDermott¹, Shirly Wang^{2

3}, Nikki Marinsek⁴, Rajesh Ranganath⁵, Luca Foschini⁴, Marzyeh Ghassemi^{2

6

7}

Affiliations

¹ Massachusetts Institute of Technology, Cambridge, MA 02139, USA. mmd@mit.edu.
² Department of Computer Science, University of Toronto, Toronto, ON M5T 3A1, Canada.
³ Layer 6 AI, TD Bank Group, Toronto, ON M5G 1M1, Canada.
⁴ Evidation Health Inc., San Mateo, CA 94402, USA.
⁵ Center for Data Science and Department of Computer Science, Courant Institute of Mathematical Sciences, New York University, New York, NY 10011, USA ; Department of Population Health, NYU Langone Health, New York, NY, USA 10016.
⁶ Vector Institute, University of Toronto, Toronto, ON M5G 1M1, Canada.
⁷ Department of Medicine, University of Toronto, Toronto, ON M5S 1A8, Canada.

PMID: 33762434
DOI: 10.1126/scitranslmed.abb1655

Review

Reproducibility in machine learning for health research: Still a ways to go

Matthew B A McDermott et al. Sci Transl Med. 2021.

. 2021 Mar 24;13(586):eabb1655.

doi: 10.1126/scitranslmed.abb1655.

Authors

Matthew B A McDermott¹, Shirly Wang^{2

3}, Nikki Marinsek⁴, Rajesh Ranganath⁵, Luca Foschini⁴, Marzyeh Ghassemi^{2

6

7}

Affiliations

¹ Massachusetts Institute of Technology, Cambridge, MA 02139, USA. mmd@mit.edu.
² Department of Computer Science, University of Toronto, Toronto, ON M5T 3A1, Canada.
³ Layer 6 AI, TD Bank Group, Toronto, ON M5G 1M1, Canada.
⁴ Evidation Health Inc., San Mateo, CA 94402, USA.
⁵ Center for Data Science and Department of Computer Science, Courant Institute of Mathematical Sciences, New York University, New York, NY 10011, USA ; Department of Population Health, NYU Langone Health, New York, NY, USA 10016.
⁶ Vector Institute, University of Toronto, Toronto, ON M5G 1M1, Canada.
⁷ Department of Medicine, University of Toronto, Toronto, ON M5S 1A8, Canada.

PMID: 33762434
DOI: 10.1126/scitranslmed.abb1655

Abstract

Machine learning for health must be reproducible to ensure reliable clinical use. We evaluated 511 scientific papers across several machine learning subfields and found that machine learning for health compared poorly to other areas regarding reproducibility metrics, such as dataset and code accessibility. We propose recommendations to address this problem.

PubMed Disclaimer

Publication types

Actions
Actions
Actions

MeSH terms

Actions
Actions

LinkOut - more resources

Full Text Sources
- Atypon
- Ovid Technologies, Inc.
Other Literature Sources
- scite Smart Citations

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Reproducibility in machine learning for health research: Still a ways to go

Affiliations

Reproducibility in machine learning for health research: Still a ways to go

Authors

Affiliations

Abstract

Publication types

MeSH terms

LinkOut - more resources

Full Text Sources

Other Literature Sources