Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 Jun 8;10(6):e035604.
doi: 10.1136/bmjopen-2019-035604.

Development of ARCADIA: a tool for assessing the quality of peer-review reports in biomedical research

Cecilia Superchi  1   2 , Darko Hren  3 , David Blanco  4   2 , Roser Rius  2 , Alessandro Recchioni  5 , Isabelle Boutron #  4   6 , José Antonio González #  2
Affiliations

Development of ARCADIA: a tool for assessing the quality of peer-review reports in biomedical research

Cecilia Superchi et al. BMJ Open. .

Abstract

Objective: To develop a tool to assess the quality of peer-review reports in biomedical research.

Methods: We conducted an online survey intended for biomedical editors and authors. The survey aimed to (1) determine if participants endorse the proposed definition of peer-review report quality; (2) identify the most important items to include in the final version of the tool and (3) identify any missing items. Participants rated on a 5-point scale whether an item should be included in the tool and they were also invited to comment on the importance and wording of each item. Principal component analysis was performed to examine items redundancy and a general inductive approach was used for qualitative data analysis.

Results: A total of 446 biomedical editors and authors participated in the survey. Participants were mainly male (65.9%), middle-aged (mean=50.3, SD=13) and with PhD degrees (56.4%). The majority of participants (84%) agreed on the definition of peer-review report quality we proposed. The 20 initial items included in the survey questionnaire were generally highly rated with a mean score ranging from 3.38 (SD=1.13) to 4.60 (SD=0.69) (scale 1-5). Participants suggested 13 items that were not included in the initial list of items. A steering committee composed of five members with different expertise discussed the selection of items to include in the final version of the tool. The final checklist includes 14 items encompassed in five domains (Importance of the study, Robustness of the study methods, Interpretation and discussion of the study results, Reporting and transparency of the manuscript, Characteristics of peer reviewer's comments).

Conclusion: Assessment of Review reports with a Checklist Available to eDItors and Authors tool could be used regularly by editors to evaluate the reviewers' work, and also as an outcome when evaluating interventions to improve the peer-review process.

Keywords: epidemiology; protocols & guidelines; public health; statistics & research methods.

PubMed Disclaimer

Conflict of interest statement

Competing interests: None declared.

Figures

Figure 1
Figure 1
Flow chart of items to include in a checklist to assess the quality of peer-review reports.
See this image and copyright information in PMC

References

    1. Stahel PF, Moore EE. Peer review for biomedical publications: we can improve the system. BMC Med 2014;12:179. 10.1186/s12916-014-0179-1 - DOI - PMC - PubMed
    1. Smith R. Peer review: a flawed process at the heart of science and journals 2006;99:5. - PMC - PubMed
    1. Rennie D. Let's make peer review scientific. Nature 2016;535:31–3. 10.1038/535031a - DOI - PubMed
    1. Jefferson T, Rudin M, Brodney Folse S, et al. . Editorial peer review for improving the quality of reports of biomedical studies. Cochrane methodology review group, editor. Cochrane Database Syst Rev 2007. - PMC - PubMed
    1. Bruce R, Chauvin A, Trinquart L, et al. . Impact of interventions to improve the quality of peer review of biomedical journals: a systematic review and meta-analysis. BMC Med 2016;14:85. 10.1186/s12916-016-0631-5 - DOI - PMC - PubMed

Publication types