Quantifying large language model usage in scientific papers

Weixin Liang^#¹, Yaohui Zhang^#², Zhengxuan Wu³, Haley Lepp⁴, Wenlong Ji⁵, Xuandong Zhao⁶, Hancheng Cao^{3

7}, Sheng Liu⁸, Siyu He⁸, Zhi Huang⁸, Diyi Yang³, Christopher Potts^{3

9}, Christopher D Manning^{3

9}, James Zou^{10

11

12}

Affiliations

¹ Department of Computer Science, Stanford University, Stanford, CA, USA. wxliang@stanford.edu.
² Department of Electrical Engineering, Stanford University, Stanford, CA, USA.
³ Department of Computer Science, Stanford University, Stanford, CA, USA.
⁴ Graduate School of Education, Stanford University, Stanford, CA, USA.
⁵ Department of Statistics, Stanford University, Stanford, CA, USA.
⁶ Department of Computer Science, University of California, Santa Barbara, Santa Barbara, CA, USA.
⁷ Goizueta Business School, Emory University, Atlanta, GA, USA.
⁸ Department of Biomedical Data Science, Stanford University, Stanford, CA, USA.
⁹ Department of Linguistics, Stanford University, Stanford, CA, USA.
¹⁰ Department of Computer Science, Stanford University, Stanford, CA, USA. jamesz@stanford.edu.
¹¹ Department of Electrical Engineering, Stanford University, Stanford, CA, USA. jamesz@stanford.edu.
¹² Department of Biomedical Data Science, Stanford University, Stanford, CA, USA. jamesz@stanford.edu.

^# Contributed equally.

PMID: 40760036
DOI: 10.1038/s41562-025-02273-8

Quantifying large language model usage in scientific papers

Weixin Liang et al. Nat Hum Behav. 2025.

. 2025 Aug 4.

doi: 10.1038/s41562-025-02273-8. Online ahead of print.

Authors

Affiliations

¹ Department of Computer Science, Stanford University, Stanford, CA, USA. wxliang@stanford.edu.
² Department of Electrical Engineering, Stanford University, Stanford, CA, USA.
³ Department of Computer Science, Stanford University, Stanford, CA, USA.
⁴ Graduate School of Education, Stanford University, Stanford, CA, USA.
⁵ Department of Statistics, Stanford University, Stanford, CA, USA.
⁶ Department of Computer Science, University of California, Santa Barbara, Santa Barbara, CA, USA.
⁷ Goizueta Business School, Emory University, Atlanta, GA, USA.
⁸ Department of Biomedical Data Science, Stanford University, Stanford, CA, USA.
⁹ Department of Linguistics, Stanford University, Stanford, CA, USA.
¹⁰ Department of Computer Science, Stanford University, Stanford, CA, USA. jamesz@stanford.edu.
¹¹ Department of Electrical Engineering, Stanford University, Stanford, CA, USA. jamesz@stanford.edu.
¹² Department of Biomedical Data Science, Stanford University, Stanford, CA, USA. jamesz@stanford.edu.

^# Contributed equally.

PMID: 40760036
DOI: 10.1038/s41562-025-02273-8

Abstract

Scientific publishing is the primary means of disseminating research findings. There has been speculation about how extensively large language models (LLMs) are being used in academic writing. Here we conduct a systematic analysis across 1,121,912 preprints and published papers from January 2020 to September 2024 on arXiv, bioRxiv and Nature portfolio journals, using a population-level framework based on word frequency shifts to estimate the prevalence of LLM-modified content over time. Our findings suggest a steady increase in LLM usage, with the largest and fastest growth estimated for computer science papers (up to 22%). By comparison, mathematics papers and the Nature portfolio showed lower evidence of LLM modification (up to 9%). LLM modification estimates were higher among papers from first authors who post preprints more frequently, papers in more crowded research areas and papers of shorter lengths. Our findings suggest that LLMs are being broadly used in scientific writing.

PubMed Disclaimer

Conflict of interest statement

Competing interests: The authors declare no competing interests.

References

1. Okunytė, P. Google search exposes academics using ChatGPT in research papers. Cybernews https://cybernews.com/news/academic-cheating-chatgpt-openai/ (2023).
1. Deguerin, M. AI-generated nonsense is leaking into scientific journals. Popular Science https://www.popsci.com/technology/ai-generated-text-scientific-journals/ (2024).
1. Oransky, I. & Marcus, A. Papers and peer reviews with evidence of ChatGPT writing. Retraction Watch https://retractionwatch.com/papers-and-peer-reviews-with-evidence-of-cha... (2024).
1. Conroy, G. Scientific sleuths spot dishonest ChatGPT use in papers. Nature https://doi.org/10.1038/d41586-023-02477-w (2023).
1. Conroy, G. How ChatGPT and other AI tools could disrupt scientific publishing. Nature https://doi.org/10.1038/d41586-023-03144-w (2023).

LinkOut - more resources

Full Text Sources
- Nature Publishing Group

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Quantifying large language model usage in scientific papers

Affiliations

Quantifying large language model usage in scientific papers

Authors

Affiliations

Abstract

Conflict of interest statement

References

LinkOut - more resources

Full Text Sources