Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Review
. 2021 Mar 24;13(586):eabb1655.
doi: 10.1126/scitranslmed.abb1655.

Reproducibility in machine learning for health research: Still a ways to go

Matthew B A McDermott  1 , Shirly Wang  2   3 , Nikki Marinsek  4 , Rajesh Ranganath  5 , Luca Foschini  4 , Marzyeh Ghassemi  2   6   7
Affiliations
Review

Reproducibility in machine learning for health research: Still a ways to go

Matthew B A McDermott et al. Sci Transl Med. .

Abstract

Machine learning for health must be reproducible to ensure reliable clinical use. We evaluated 511 scientific papers across several machine learning subfields and found that machine learning for health compared poorly to other areas regarding reproducibility metrics, such as dataset and code accessibility. We propose recommendations to address this problem.

PubMed Disclaimer

Figures

Figure 1:
Figure 1:
Fraction of papers satisfying certain conditions by ML field. See the Supplementary Material for detailed descriptions of the underlying data collection procedure. Note that ML4H consistently lags other subfields of machine learning on all measures of reproducibility save inclusion of proper statistical variance.
Figure 2:
Figure 2:
Summary of recommendations for different stakeholders.
See this image and copyright information in PMC

References

    1. Baker M, 1,500 scientists lift the lid on reproducibility, Nature News 533, 452 (2016). - PubMed
    1. Cokol M, Ozbay F, RodriguezEsteban R, Retraction rates are on the rise, EMBO reports 9, 2 (2008). - PMC - PubMed
    1. Gundersen OE, Kjensmo S, State of the art: Reproducibility in artificial intelligence, Thirty-Second AAAI Conference on Artificial Intelligence (2018).
    1. Beam AL, Manrai AK, Ghassemi M, Challenges to the reproducibility of machine learning models in health care, JAMA (2020). - PMC - PubMed
    1. Nagendran M, et al. , Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies, bmj 368 (2020). - PMC - PubMed

Publication types

MeSH terms

LinkOut - more resources