Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Mar;25(2):254-263.
doi: 10.5811/westjem.18432.

Development and Validation of a Scoring Rubric for Editorial Evaluation of Peer-review Quality: A Pilot Study

Jeffrey N Love  1 , Anne M Messman  2 , Jonathan S Ilgen  3 , Chris Merritt  4   5 , Wendy C Coates  6 , Douglas S Ander  7 , David P Way  8
Affiliations

Development and Validation of a Scoring Rubric for Editorial Evaluation of Peer-review Quality: A Pilot Study

Jeffrey N Love et al. West J Emerg Med. 2024 Mar.

Abstract

Introduction: Despite the importance of peer review to publications, there is no generally accepted approach for editorial evaluation of a peer review's value to a journal editor's decision-making. The graduate medical education editors of the Western Journal of Emergency Medicine Special Issue in Educational Research & Practice (Special Issue) developed and studied the holistic editor's scoring rubric (HESR) with the objective of assessing the quality of a review and an emphasis on the degree to which it informs a holistic appreciation for the submission under consideration.

Methods: Using peer-review guidelines from several journals, the Special Issue's editors formulated the rubric as descriptions of peer reviews of varying degree of quality from the ideal to the unacceptable. Once a review was assessed by each editor using the rubric, the score was submitted to a third party for blinding purposes. We compared the performance of the new rubric to a previously used semantic differential scale instrument. Kane's validity framework guided the evaluation of the new scoring rubric around three basic assumptions: improved distribution of scores; relative consistency rather than absolute inter-rater reliability across editors; and statistical evidence that editors valued peer reviews that contributed most to their decision-making.

Results: Ninety peer reviews were the subject of this study, all were assessed by two editors. Compared to the highly skewed distribution of the prior rating scale, the distribution of the new scoring rubric was bell shaped and demonstrated full use of the rubric scale. Absolute agreement between editors was low to moderate, while relative consistency between editor's rubric ratings was high. Finally, we showed that recommendations of higher rated peer reviews were more likely to concur with the editor's formal decision.

Conclusion: Early evidence regarding the HESR supports the use of this instrument in determining the quality of peer reviews as well as its relative importance in informing editorial decision-making.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

Figures

Figure 1.
Figure 1.
The holistic editor’s scoring rubric used for evaluation of peer review. The initial version of this rubric can be appreciated as the unshaded content. Subsequent additions made based on a pilot of the 14 initial reviews are denoted by the shaded areas (See “Preliminary Calibration Exercise.”) CORD, Council of Residency Directors in Emergency Medicine.
Figure 2.
Figure 2.
Flow chart showing the number of manuscripts submitted and processed during the 2020–21 submissions cycle for the Western Journal of Emergency Medicine Special Issue in Educational Research & Practice. UGME, undergraduate medical education; CDEM, Clerkship Directors of Emergency Medicine; GME, graduate medical education; CORD, Council of Residency Directors in Emergency Medicine.
Figure 3.
Figure 3.
Comparison of the peer-review ratings distributions for two methods of editor evaluations across two Special Issue production cycles. The first method involved a 5-point semantic differential scale with labels only at the end points, which was used during the 2020 cycle. The second method involved the holistic editor’s scoring rubric used by the CORD editors at the pilot and full implementation stages of the 2021 production cycle. CORD, Council of Residency Directors in Emergency Medicine. *Data significantly deviate from normal distribution (P ≤ 0.001).
See this image and copyright information in PMC

References

    1. Lock S. A Difficult Balance: Editorial Peer Review in Medicine. Philadelphia, PA: ISI Press, 1986.1–38,122–132.
    1. Cicchetti DV. The reliability of peer review for manuscript and grant submissions: a cross-disciplinary investigation. Behav Brain Sci. 1991;14(1):119–35.
    1. Strayhorn J, Jr, McDermott JF, Jr, Tanguay P. An intervention to improve the reliability of manuscript reviews for the Journal of the American Academy of Child and Adolescent Psychiatry. Am J of Psychiatry. 1993;150(6):947–52. - PubMed
    1. Howard L, Wilkinson G. Peer review and editorial decision making. Br J Psychiatry. 1998;173:110–3. - PubMed
    1. Callaham ML, Baxt WG, Waeckerle JF, et al. . Reliability of editors’ subjective quality ratings of peer reviews of manuscripts. JAMA. 1998;280(3):229–31. - PubMed