. 2023 Aug 1;6(8):e2331410.

doi: 10.1001/jamanetworkopen.2023.31410.

Characterization of Comments About bioRxiv and medRxiv Preprints

Clarissa França Dias Carneiro^{1

2}, Gabriel Gonçalves da Costa¹, Kleber Neves¹, Mariana Boechat Abreu³, Pedro Batista Tan^{1

4

5

6}, Danielle Rayêe⁷, Flávia Zacouteguy Boos⁸, Roberta Andrejew⁹, Tiago Lubiana^{10

11}, Mario Malicki^{12

13

14}, Olavo Bohrer Amaral¹

Affiliations

¹ Institute of Medical Biochemistry Leopoldo de Meis, Universidade Federal do Rio de Janeiro, Rio de Janeiro, Brazil.
² Berlin Institute of Health at Charité-Universitätsmedizin Berlin, QUEST Center for Responsible Research, Berlin, Germany.
³ Carlos Chagas Filho Institute of Biophysics, Universidade Federal do Rio de Janeiro, Rio de Janeiro, Brazil.
⁴ Department of Gastrointestinal Oncology, Netherlands Cancer Institute, Amsterdam, the Netherlands.
⁵ Division of Molecular Carcinogenesis, Netherlands Cancer Institute, Amsterdam, the Netherlands.
⁶ Oncode Institute, Utrecht, the Netherlands.
⁷ Department of Ophthalmology and Visual Sciences, Albert Einstein College of Medicine, Bronx, New York.
⁸ Programa de Pós-graduação em Psicobiologia, Universidade Federal de São Paulo, São Paulo, Brazil.
⁹ Department of Biochemistry, Institute of Chemistry, Universidade de São Paulo, São Paulo, Brazil.
¹⁰ Ronin Institute, Virtual Organization, São Paulo, Brazil.
¹¹ School of Pharmaceutical Sciences, University of São Paulo, São Paulo, Brazil.
¹² Department of Epidemiology and Population Health, Stanford University, Stanford, California.
¹³ Stanford Program on Research Rigor and Reproducibility, Stanford University, Stanford, California.
¹⁴ Meta-Research Innovation Center at Stanford, Stanford University, Stanford, California.

PMID: 37647065
PMCID: PMC10469270
DOI: 10.1001/jamanetworkopen.2023.31410

Characterization of Comments About bioRxiv and medRxiv Preprints

Clarissa França Dias Carneiro et al. JAMA Netw Open. 2023.

. 2023 Aug 1;6(8):e2331410.

doi: 10.1001/jamanetworkopen.2023.31410.

Authors

Affiliations

¹ Institute of Medical Biochemistry Leopoldo de Meis, Universidade Federal do Rio de Janeiro, Rio de Janeiro, Brazil.
² Berlin Institute of Health at Charité-Universitätsmedizin Berlin, QUEST Center for Responsible Research, Berlin, Germany.
³ Carlos Chagas Filho Institute of Biophysics, Universidade Federal do Rio de Janeiro, Rio de Janeiro, Brazil.
⁴ Department of Gastrointestinal Oncology, Netherlands Cancer Institute, Amsterdam, the Netherlands.
⁵ Division of Molecular Carcinogenesis, Netherlands Cancer Institute, Amsterdam, the Netherlands.
⁶ Oncode Institute, Utrecht, the Netherlands.
⁷ Department of Ophthalmology and Visual Sciences, Albert Einstein College of Medicine, Bronx, New York.
⁸ Programa de Pós-graduação em Psicobiologia, Universidade Federal de São Paulo, São Paulo, Brazil.
⁹ Department of Biochemistry, Institute of Chemistry, Universidade de São Paulo, São Paulo, Brazil.
¹⁰ Ronin Institute, Virtual Organization, São Paulo, Brazil.
¹¹ School of Pharmaceutical Sciences, University of São Paulo, São Paulo, Brazil.
¹² Department of Epidemiology and Population Health, Stanford University, Stanford, California.
¹³ Stanford Program on Research Rigor and Reproducibility, Stanford University, Stanford, California.
¹⁴ Meta-Research Innovation Center at Stanford, Stanford University, Stanford, California.

PMID: 37647065
PMCID: PMC10469270
DOI: 10.1001/jamanetworkopen.2023.31410

Abstract

Importance: Preprints have been increasingly used in biomedical science, and a key feature of many platforms is public commenting. The content of these comments, however, has not been well studied, and it is unclear whether they resemble those found in journal peer review.

Objective: To describe the content of comments on the bioRxiv and medRxiv preprint platforms.

Design, setting, and participants: In this cross-sectional study, preprints posted on the bioRxiv and medRxiv platforms in 2020 were accessed through each platform's application programming interface on March 29, 2021, and a random sample of preprints containing between 1 and 20 comments was evaluated independently by 3 evaluators using an instrument to assess their features and general content.

Main outcome and measures: The numbers and percentages of comments from authors or nonauthors were assessed, and the comments from nonauthors were assessed for content. These nonauthor comments were assessed to determine whether they included compliments, criticisms, corrections, suggestions, or questions, as well as their topics (eg, relevance, interpretation, and methods). Nonauthor comments were also analyzed to determine whether they included references, provided a summary of the findings, or questioned the preprint's conclusions.

Results: Of 52 736 preprints, 3850 (7.3%) received at least 1 comment (mean [SD] follow-up, 7.5 [3.6] months), and the 1921 assessed comments (from 1037 preprints) had a median length of 43 words (range, 1-3172 words). The criticisms, corrections, or suggestions present in 694 of 1125 comments (61.7%) were the most prevalent content, followed by compliments (n = 428 [38.0%]) and questions (n = 393 [35.0%]). Criticisms usually regarded interpretation (n = 286), methodological design (n = 267), and data collection (n = 238), while compliments were mainly about relevance (n = 111) and implications (n = 72).

Conclusions and relevance: In this cross-sectional study of preprint comments, topics commonly associated with journal peer review were frequent. However, only a small percentage of preprints posted on the bioRxiv and medRxiv platforms in 2020 received comments on these platforms. A clearer taxonomy of peer review roles would help to describe whether postpublication peer review fulfills them.

PubMed Disclaimer

Conflict of interest statement

Conflict of Interest Disclosures: Dr Neves reported being a board member at ASAPbio, a nonprofit organization promoting the productive use of preprints in the life sciences. Dr Malički reported being an Editor in Chief of Research Integrity and Peer Review. Dr Amaral reported receiving grants from Fundação de Amparo à Pesquisa do Estado do Rio de Janeiro (FAPERJ) covering part of the current work, personal fees from a CNPq Research Productivity scholarship during the conduct of the study, and grants from eLife outside the submitted work and has consistently advocated for the use of preprints over the past 5 years, having acted as a pro bono ambassador for ASAPbio in the past. No other disclosures were reported.

Figures

**Figure 1.. Flow Diagram of the Analysis Process**
Responses to other comments were not analyzed. Comments from the preprint authors and those classified as not about the preprint content were analyzed using specific categories. All other comments (ie, not responses, from nonauthors, and about the content) were analyzed using the main data extraction form.

**Figure 2.. Sample Description**
Bars represent the total number of preprints in each bin (ie, those from bioRxiv and those from medRxiv). A, Number of comments per sampled preprint. The median is 1, with a maximum of 17 for bioRxiv and 18 for medRxiv (those with >20 comments were excluded). B, Distribution of comment length. The median length is 42 words (range, 1-3172 words) on the bioRxiv platform and 44 words (range, 1-1640 words) on the medRxiv platform; the overall median length is 43 words (range, 1-1640 words). The peak in 1-word comments mostly consists of isolated hyperlinks. C, Distribution of number of comments by publication month.

**Figure 3.. Specific Content of Comments**
A, Specific content of criticisms, corrections, or suggestions. B, Specific content of compliments. C, Specific content of questions. Categories are not mutually exclusive (ie, each comment could include multiple categories); thus, percentages do not add up to 100%.

See this image and copyright information in PMC

References

1. Berg JM, Bhalla N, Bourne PE, et al. . Preprints for the life sciences. Science. 2016;352(6288):899-901. doi:10.1126/science.aaf9133 - DOI - PubMed
1. Biology preprints over time. ASAPbio. Accessed March 3, 2023. https://asapbio.org/preprint-info/biology-preprints-over-time
1. Fraser N, Brierley L, Dey G, et al. . The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape. PLoS Biol. 2021;19(4):e3000959. doi:10.1371/journal.pbio.3000959 - DOI - PMC - PubMed
1. Sever R, Roeder T, Hindle S, et al. . bioRxiv: the preprint server for biology. bioRxiv. Preprint posted online November 6, 2019. doi:10.1101/833400 - DOI
1. Dolgin E. PubMed Commons closes its doors to comments. Nature. February 2, 2018. doi:10.1038/d41586-018-01591-4 - DOI

Publication types

Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions

LinkOut - more resources

Full Text Sources

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Characterization of Comments About bioRxiv and medRxiv Preprints

Affiliations

Characterization of Comments About bioRxiv and medRxiv Preprints

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

References

Publication types

MeSH terms

LinkOut - more resources

Full Text Sources