REFORMS: Consensus-based Recommendations for Machine-learning-based Science

doi:10.1126/sciadv.adk3452

Review

. 2024 May 3;10(18):eadk3452.

doi: 10.1126/sciadv.adk3452. Epub 2024 May 1.

REFORMS: Consensus-based Recommendations for Machine-learning-based Science

Sayash Kapoor^{1

2}, Emily M Cantrell^{3

4}, Kenny Peng⁵, Thanh Hien Pham^{1

2}, Christopher A Bail^{6

7

8}, Odd Erik Gundersen^{9

10}, Jake M Hofman¹¹, Jessica Hullman¹², Michael A Lones¹³, Momin M Malik^{14

15

16}, Priyanka Nanayakkara^{12

17}, Russell A Poldrack¹⁸, Inioluwa Deborah Raji¹⁹, Michael Roberts^{20

21}, Matthew J Salganik^{2

3

22}, Marta Serra-Garcia²³, Brandon M Stewart^{2

3

22

24}, Gilles Vandewiele²⁵, Arvind Narayanan^{1

2}

Affiliations

¹ Department of Computer Science, Princeton University, Princeton, NJ 08544, USA.
² Center for Information Technology Policy, Princeton University, Princeton, NJ 08544, USA.
³ Department of Sociology, Princeton University, Princeton, NJ 08544, USA.
⁴ School of Public and International Affairs, Princeton University, Princeton, NJ 08544, USA.
⁵ Department of Computer Science, Cornell University, Ithaca, NY 14850, USA.
⁶ Department of Sociology, Duke University, Durham, NC 27708, USA.
⁷ Department of Political Science, Duke University, Durham, NC 27708, USA.
⁸ Sanford School of Public Policy, Duke University, Durham, NC 27708, USA.
⁹ Department of Computer Science, Norwegian University of Science and Technology, Trondheim, Norway.
¹⁰ Aneo AS, Trondheim, Norway.
¹¹ Microsoft Research, New York, NY 10012, USA.
¹² Department of Computer Science, Northwestern University, Evanston, IL 60208, USA.
¹³ School of Mathematical and Computer Sciences, Heriot-Watt University, Edinburgh, UK.
¹⁴ Center for Digital Health, Mayo Clinic, Rochester, MN 55905, USA.
¹⁵ School of Social Policy & Practice, University of Pennsylvania, Philadelphia, PA 19104, USA.
¹⁶ Institute in Critical Quantitative, Computational, & Mixed Methodologies, Johns Hopkins University, Baltimore, MD 21218, USA.
¹⁷ Department of Communication Studies, Northwestern University, Evanston, IL 60208, USA.
¹⁸ Department of Psychology, Stanford University, Stanford, CA 94305, USA.
¹⁹ Department of Computer Science, University of California, Berkeley, Berkeley, CA 94720, USA.
²⁰ Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK.
²¹ Department of Medicine, University of Cambridge, Cambridge, UK.
²² Office of Population Research, Princeton University, Princeton, NJ 08544, USA.
²³ Rady School of Management, University of California, San Diego, La Jolla, CA 92093, USA.
²⁴ Department of Politics, Princeton University, Princeton, NJ 08544, USA.
²⁵ Department of Information Technology, Ghent University, Ghent, Belgium.

PMID: 38691601
PMCID: PMC11092361
DOI: 10.1126/sciadv.adk3452

Review

REFORMS: Consensus-based Recommendations for Machine-learning-based Science

Sayash Kapoor et al. Sci Adv. 2024.

. 2024 May 3;10(18):eadk3452.

doi: 10.1126/sciadv.adk3452. Epub 2024 May 1.

Authors

Affiliations

¹ Department of Computer Science, Princeton University, Princeton, NJ 08544, USA.
² Center for Information Technology Policy, Princeton University, Princeton, NJ 08544, USA.
³ Department of Sociology, Princeton University, Princeton, NJ 08544, USA.
⁴ School of Public and International Affairs, Princeton University, Princeton, NJ 08544, USA.
⁵ Department of Computer Science, Cornell University, Ithaca, NY 14850, USA.
⁶ Department of Sociology, Duke University, Durham, NC 27708, USA.
⁷ Department of Political Science, Duke University, Durham, NC 27708, USA.
⁸ Sanford School of Public Policy, Duke University, Durham, NC 27708, USA.
⁹ Department of Computer Science, Norwegian University of Science and Technology, Trondheim, Norway.
¹⁰ Aneo AS, Trondheim, Norway.
¹¹ Microsoft Research, New York, NY 10012, USA.
¹² Department of Computer Science, Northwestern University, Evanston, IL 60208, USA.
¹³ School of Mathematical and Computer Sciences, Heriot-Watt University, Edinburgh, UK.
¹⁴ Center for Digital Health, Mayo Clinic, Rochester, MN 55905, USA.
¹⁵ School of Social Policy & Practice, University of Pennsylvania, Philadelphia, PA 19104, USA.
¹⁶ Institute in Critical Quantitative, Computational, & Mixed Methodologies, Johns Hopkins University, Baltimore, MD 21218, USA.
¹⁷ Department of Communication Studies, Northwestern University, Evanston, IL 60208, USA.
¹⁸ Department of Psychology, Stanford University, Stanford, CA 94305, USA.
¹⁹ Department of Computer Science, University of California, Berkeley, Berkeley, CA 94720, USA.
²⁰ Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK.
²¹ Department of Medicine, University of Cambridge, Cambridge, UK.
²² Office of Population Research, Princeton University, Princeton, NJ 08544, USA.
²³ Rady School of Management, University of California, San Diego, La Jolla, CA 92093, USA.
²⁴ Department of Politics, Princeton University, Princeton, NJ 08544, USA.
²⁵ Department of Information Technology, Ghent University, Ghent, Belgium.

PMID: 38691601
PMCID: PMC11092361
DOI: 10.1126/sciadv.adk3452

Abstract

Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear recommendations for conducting and reporting ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist (recommendations for machine-learning-based science). It consists of 32 questions and a paired set of guidelines. REFORMS was developed on the basis of a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility.

PubMed Disclaimer

Cited by

Hierarchical, Interactive, and Dynamic Predictive Capacity of Current Biological, Psychological, Social, and Environmental Measurements in Depression, Anxiety, ADHD, and Social Quality across the Lifespan.
Roberts C, Yamamoto R, Fawnia Z, Mistry P, Treves IN, Decker A, Sallie SN, Chatham C, Aftab A, Lee K, Park M, Menon V, Gabrieli J. Roberts C, et al. Res Sq [Preprint]. 2025 Jul 30:rs.3.rs-7060126. doi: 10.21203/rs.3.rs-7060126/v1. Res Sq. 2025. PMID: 40766227 Free PMC article. Preprint.
The reanimation of pseudoscience in machine learning and its ethical repercussions.
Andrews M, Smart A, Birhane A. Andrews M, et al. Patterns (N Y). 2024 Aug 1;5(9):101027. doi: 10.1016/j.patter.2024.101027. eCollection 2024 Sep 13. Patterns (N Y). 2024. PMID: 39568649 Free PMC article. Review.
Unraveling overoptimism and publication bias in ML-driven science.
Saidi P, Dasarathy G, Berisha V. Saidi P, et al. Patterns (N Y). 2025 Feb 25;6(4):101185. doi: 10.1016/j.patter.2025.101185. eCollection 2025 Apr 11. Patterns (N Y). 2025. PMID: 40264959 Free PMC article.
Why an overreliance on AI-driven modelling is bad for science.
Narayanan A, Kapoor S. Narayanan A, et al. Nature. 2025 Apr;640(8058):312-314. doi: 10.1038/d41586-025-01067-2. Nature. 2025. PMID: 40195515 No abstract available.
Review and recommendations for using artificial intelligence in intracoronary optical coherence tomography analysis.
Chen X, Huang Y, Jessney B, Sangha J, Gu S, Schönlieb CB, Bennett M, Roberts M. Chen X, et al. Eur Heart J Digit Health. 2025 May 15;6(4):529-539. doi: 10.1093/ehjdh/ztaf053. eCollection 2025 Jul. Eur Heart J Digit Health. 2025. PMID: 40703128 Free PMC article. Review.

See all "Cited by" articles

References

1. Athey S., Imbens G. W., Machine learning methods that economists should know about. Annu. Rev. Econ. 11, 685–725 (2019).
1. Schrider D. R., Kern A. D., Supervised machine learning for population genetics: A new paradigm. Trends Genet. 34, 301–312 (2018). - PMC - PubMed
1. Valletta J. J., Torney C., Kings M., Thornton A., Madden J., Applications of machine learning in animal behaviour studies. Anim. Behav. 124, 203–220 (2017).
1. Iniesta R., Stahl D., McGuffin P., Machine learning, statistical learning and the future of biological research in psychiatry. Psychol. Med. 46, 2455–2465 (2016). - PMC - PubMed
1. Tonidandel S., King E. B., Cortina J. M., Big data methods: Leveraging modern data analytic techniques to build organizational science. Organ. Res. Methods 21, 525–547 (2018).

Publication types

Actions
Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions

Grants and funding

P2C HD065563/HD/NICHD NIH HHS/United States

LinkOut - more resources

Full Text Sources
Miscellaneous
- NCI CPTAC Assay Portal

[1] Athey S., Imbens G. W., Machine learning methods that economists should know about. Annu. Rev. Econ. 11, 685–725 (2019).

[2] Athey S., Imbens G. W., Machine learning methods that economists should know about. Annu. Rev. Econ. 11, 685–725 (2019).

[3] Schrider D. R., Kern A. D., Supervised machine learning for population genetics: A new paradigm. Trends Genet. 34, 301–312 (2018). - PMC - PubMed

[4] Schrider D. R., Kern A. D., Supervised machine learning for population genetics: A new paradigm. Trends Genet. 34, 301–312 (2018). - PMC - PubMed

[5] Valletta J. J., Torney C., Kings M., Thornton A., Madden J., Applications of machine learning in animal behaviour studies. Anim. Behav. 124, 203–220 (2017).

[6] Valletta J. J., Torney C., Kings M., Thornton A., Madden J., Applications of machine learning in animal behaviour studies. Anim. Behav. 124, 203–220 (2017).

[7] Iniesta R., Stahl D., McGuffin P., Machine learning, statistical learning and the future of biological research in psychiatry. Psychol. Med. 46, 2455–2465 (2016). - PMC - PubMed

[8] Iniesta R., Stahl D., McGuffin P., Machine learning, statistical learning and the future of biological research in psychiatry. Psychol. Med. 46, 2455–2465 (2016). - PMC - PubMed

[9] Tonidandel S., King E. B., Cortina J. M., Big data methods: Leveraging modern data analytic techniques to build organizational science. Organ. Res. Methods 21, 525–547 (2018).

[10] Tonidandel S., King E. B., Cortina J. M., Big data methods: Leveraging modern data analytic techniques to build organizational science. Organ. Res. Methods 21, 525–547 (2018).

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

REFORMS: Consensus-based Recommendations for Machine-learning-based Science

Affiliations

REFORMS: Consensus-based Recommendations for Machine-learning-based Science

Authors

Affiliations

Abstract

Similar articles

Cited by

References

Publication types

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources

Miscellaneous

Abstract

Similar articles

Cited by

References

Publication types

MeSH terms

Related information

Grants and funding

LinkOut - more resources

Full Text Sources

Miscellaneous