Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2010 Jun 28;5(6):e11344.
doi: 10.1371/journal.pone.0011344.

The usefulness of peer review for selecting manuscripts for publication: a utility analysis taking as an example a high-impact journal

Lutz Bornmann  1 , Hans-Dieter Daniel
Affiliations

The usefulness of peer review for selecting manuscripts for publication: a utility analysis taking as an example a high-impact journal

Lutz Bornmann et al. PLoS One. .

Abstract

Background: High predictive validity--that is, a strong association between the outcome of peer review (usually, reviewers' ratings) and the scientific quality of a manuscript submitted to a journal (measured as citations of the later published paper)--does not as a rule suffice to demonstrate the usefulness of peer review for the selection of manuscripts. To assess usefulness, it is important to include in addition the base rate (proportion of submissions that are fundamentally suitable for publication) and the selection rate (the proportion of submissions accepted).

Methodology/principal findings: Taking the example of the high-impact journal Angewandte Chemie International Edition (AC-IE), we present a general approach for determining the usefulness of peer reviews for the selection of manuscripts for publication. The results of our study show that peer review is useful: 78% of the submissions accepted by AC-IE are correctly accepted for publication when the editor's decision is based on one review, 69% of the submissions are correctly accepted for publication when the editor's decision is based on two reviews, and 65% of the submissions are correctly accepted for publication when the editor's decision is based on three reviews.

Conclusions/significance: The paper points out through what changes in the selection rate, base rate or validity coefficient a higher success rate (utility) in the AC-IE selection process could be achieved.

PubMed Disclaimer

Conflict of interest statement

Competing Interests: The authors have declared that no competing interests exist.

Figures

Figure 1
Figure 1. Dependency of the number of accepted and qualified submissions (AQ) on base rate, selection rate and validity coefficient (rxy).
Figure 2
Figure 2. Base rate, selection rate, success rate, and validity coefficient (rxy) in the reviewing of AC-IE submissions by one (Graph A), two (Graph B), or three (Graph C) reviewers.
The green line in the graphs along with 95% confidence interval (gray lines) is the prediction for the criterion, based on a linear regression of the criterion on the predictor.
See this image and copyright information in PMC

References

    1. Sense About Science. Peer review and the acceptance of new scientific ideas. London, UK: Sense about Science (working party on equipping the public with an understanding of peer review); 2004.
    1. Stossel TP. Refinement in biomedical communication - a case study. Science Technology & Human Values. 1985;10:39–43.
    1. Bornmann L, Daniel H-D. Selecting manuscripts for a high impact journal through peer review: a citation analysis of Communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere. Journal of the American Society for Information Science and Technology. 2008;59:1841–1852.
    1. Bornmann L, Daniel H-D. The effectiveness of the peer review process: inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angewandte Chemie International Edition. 2008;47:7173–7178. - PubMed
    1. Bornmann L, Daniel H-D. Extent of type I and type II errors in editorial decisions: a case study on Angewandte Chemie International Edition. Journal of Informetrics. 2009;3:348–352.

Publication types