Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2010 Apr 8;5(4):e10072.
doi: 10.1371/journal.pone.0010072.

Editorial peer reviewers' recommendations at a general medical journal: are they reliable and do editors care?

Richard L Kravitz  1 , Peter Franks, Mitchell D Feldman, Martha Gerrity, Cindy Byrne, William M Tierney
Affiliations

Editorial peer reviewers' recommendations at a general medical journal: are they reliable and do editors care?

Richard L Kravitz et al. PLoS One. .

Abstract

Background: Editorial peer review is universally used but little studied. We examined the relationship between external reviewers' recommendations and the editorial outcome of manuscripts undergoing external peer-review at the Journal of General Internal Medicine (JGIM).

Methodology/principal findings: We examined reviewer recommendations and editors' decisions at JGIM between 2004 and 2008. For manuscripts undergoing peer review, we calculated chance-corrected agreement among reviewers on recommendations to reject versus accept or revise. Using mixed effects logistic regression models, we estimated intra-class correlation coefficients (ICC) at the reviewer and manuscript level. Finally, we examined the probability of rejection in relation to reviewer agreement and disagreement. The 2264 manuscripts sent for external review during the study period received 5881 reviews provided by 2916 reviewers; 28% of reviews recommended rejection. Chance corrected agreement (kappa statistic) on rejection among reviewers was 0.11 (p<.01). In mixed effects models adjusting for study year and manuscript type, the reviewer-level ICC was 0.23 (95% confidence interval [CI], 0.19-0.29) and the manuscript-level ICC was 0.17 (95% CI, 0.12-0.22). The editors' overall rejection rate was 48%: 88% when all reviewers for a manuscript agreed on rejection (7% of manuscripts) and 20% when all reviewers agreed that the manuscript should not be rejected (48% of manuscripts) (p<0.01).

Conclusions/significance: Reviewers at JGIM agreed on recommendations to reject vs. accept/revise at levels barely beyond chance, yet editors placed considerable weight on reviewers' recommendations. Efforts are needed to improve the reliability of the peer-review process while helping editors understand the limitations of reviewers' recommendations.

PubMed Disclaimer

Conflict of interest statement

Competing Interests: Drs. Kravitz and Feldman are the current editors-in-chief of the Journal of General Internal Medicine. Drs. Tierney and Gerrity are the immediate past editors-in-chief. Ms. Byrne is managing editor of the journal. This does not alter the authors' adherence to all the PLoS ONE policies on sharing data and materials.

Figures

Figure 1
Figure 1. Flow chart showing outcome of reviews pertaining to 2264 manuscripts undergoing external peer review at the Journal of General Internal Medicine.
See this image and copyright information in PMC

References

    1. Relman AS. Peer review in scientific journals–what good is it? West J Med. 1990;153(5):520–2. - PMC - PubMed
    1. Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Ann Intern Med. 1994;121(1):11–21. - PubMed
    1. Tierney W, Gerrity M. Thanks to Reviewers. J Gen Intern Med. 2008;23(12):2145–54.
    1. Hojat M, Gonnella JS, Caelleigh AS. Impartial judgment by the “gatekeepers” of science: fallibility and accountability in the peer review process. Adv Health Sci Educ Theory Pract. 2003;8(1):75–96. - PubMed
    1. Cullen DJ, Macaulay A. Consistency between peer reviewers for a clinical specialty journal. Acad Med. 1992;67(12):856–9. - PubMed

Publication types

MeSH terms