Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2007 Apr 18;2007(2):MR000016.
doi: 10.1002/14651858.MR000016.pub3.

Editorial peer review for improving the quality of reports of biomedical studies

T Jefferson, M Rudin, S Brodney Folse, F Davidoff

Editorial peer review for improving the quality of reports of biomedical studies

T Jefferson et al. Cochrane Database Syst Rev. .

Abstract

Background: Scientific findings must withstand critical review if they are to be accepted as valid, and editorial peer review (critique, effort to disprove) is an essential element of the scientific process. We review the evidence of the editorial peer-review process of original research studies submitted for paper or electronic publication in biomedical journals.

Objectives: To estimate the effect of processes in editorial peer review.

Search strategy: The following databases were searched to June 2004: CINAHL, Ovid, Cochrane Methodology Register, Dissertation abstracts, EMBASE, Evidence Based Medicine Reviews: ACP Journal Club, MEDLINE, PsycINFO, PubMed.

Selection criteria: We included prospective or retrospective comparative studies with two or more comparison groups, generated by random or other appropriate methods, and reporting original research, regardless of publication status. We hoped to find studies identifying good submissions on the basis of: importance of the topic dealt with, relevance of the topic to the journal, usefulness of the topic, soundness of methods, soundness of ethics, completeness and accuracy of reporting.

Data collection and analysis: Because of the diversity of study questions, viewpoints, methods, and outcomes, we carried out a descriptive review of included studies grouping them by broad study question.

Main results: We included 28 studies. We found no clear-cut evidence of effect of the well-researched practice of reviewer and/or author concealment on the outcome of the quality assessment process (9 studies). Checklists and other standardisation media have some evidence to support their use (2 studies). There is no evidence that referees' training has any effect on the quality of the outcome (1 study). Different methods of communicating with reviewers and means of dissemination do not appear to have an effect on quality (3 studies). On the basis of one study, little can be said about the ability of the peer-review process to detect bias against unconventional drugs. Validity of peer review was tested by only one small study in a specialist area. Editorial peer review appears to make papers more readable and improve the general quality of reporting (2 studies), but the evidence for this has very limited generalisability.

Authors' conclusions: At present, little empirical evidence is available to support the use of editorial peer review as a mechanism to ensure quality of biomedical research. However, the methodological problems in studying peer review are many and complex. At present, the absence of evidence on efficacy and effectiveness cannot be interpreted as evidence of their absence. A large, well-funded programme of research on the effects of editorial peer review should be urgently launched.

PubMed Disclaimer

Conflict of interest statement

Tom Jefferson co‐edited the book "Peer review in health sciences". All authors are active peer reviewers and have published articles in peer‐reviewed journals.

Update of

  • doi: 10.1002/14651858.MR000016.pub2

References

References to studies included in this review

Arnau 2003 {published data only}
    1. Arnau C, Cobo E, Ribera JM, Cardellach F, Selva A, Urrutia A. [[Efecto de la revision estadistìca en la calidad de los manuscritos publicados en Medicina Clìnica: estudio aleatorizado]. Med Clin (Barc) 2003;121(18):690‐4]. - PubMed
Bingham 1998 {published data only}
    1. Bingham CM, Higgins G, Coleman R, Weyden MB. The Medical Journal of Australia internet peer‐review study. The Lancet 1998;352:441‐5. - PubMed
Callaham 1998 {published data only}
    1. Callaham ML, Wears RL, Waeckerle JF. Effect of attendance at a training session on peer reviewer quality and performance. Annals of Emergency Medicine 1998;32(3):318‐22. - PubMed
Callaham 2002 {published data only}
    1. Callaham ML, Schriger DL. Effect of structured workshop training on subsequent performance of journal peer reviews. Annals of emergency medicine 2002;40(3):323‐28. - PubMed
Callaham 2002a {published data only}
    1. Callaham ML, Knopp RK, Gallagher EJ. Effect of written feedback by editors on quality of reviews: Two randomized trials. Journal of the American Medical Association 2002;287(21):2781‐2783. - PubMed
Das Sinha 1999 {published data only}
    1. Das Sinha S, Sahni P, Nundy S. Does exchanging comments of Indian and non‐Indian reviewers improve the quality of manuscript reviews?. The National Medical Journal of India 1999;12(5):210‐3. - PubMed
Day 2002 {published data only}
    1. Day FC, Schriger DL, Todd C, Wears RL. The use of dedicated methodology and statistical reviewers for peer review: a content analysis of comments to authors made by methodology and regular reviewers. Annals of Emergency Medicine 2002;40(3):329‐33. - PubMed
Elvik 1998 {published data only}
    1. Elvik R. Are road safety evaluation studies published in peer reviewed journals more valid than similar studies not published in peer reviewed journals?. Accid Anal and Prev 1998;30(1):101‐18. - PubMed
Ernst 1999 {published data only}
    1. Ernst E, Resch K‐L. Reviewer bias against the unconventional? A randomized double‐blind study of peer review. Complementary Therapies in Medicine 1999;7:19‐23. - PubMed
Fisher 1994 {published data only}
    1. Fisher M, Friedman SB, Strauss B. The effects of blinding on acceptance of research papers by peer review. JAMA 1994;272:143‐6. - PubMed
Gardner 1990 {published data only}
    1. Gardner MJ, Bond J. An exploratory study of statistical assessment of papers published in the British Medical Journal. JAMA 1990;263(10):1355‐7. - PubMed
Godlee 1998 {published data only}
    1. Godlee F, Gale CR, Martyn CN. Effect on the quality of peer review of blinding peer reviewers and asking them to sign their reports. JAMA 1998;280(3):237‐40. - PubMed
Goodman 1994 {published data only}
    1. Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine.. Ann Intern.Med. 1994;121:11‐21.. - PubMed
Jadad 1996 {published data only}
    1. Jadad AR, Moore A, Carroll D, Jenkinson C, Reynolds DJM, Gavaghan DJ, McQuay HJ. Assessing the quality of reports of randomized clinical trials: is blinding necessary?. Controlled Clinical Trials 1996;17:1‐12. - PubMed
Jefferson 1998 {published data only}
    1. Jefferson T, Smith R, Yee Y, Drummond M, Prat M, Gale R. Evaluating the BMJ guidelines for economic submissions: prospective audit of economic submissions to BMJ and The Lancet. JAMA 1998;280(3):275‐7. - PubMed
Justice 1998 {published data only}
    1. Justice AC, Cho MK, Winker MA, Berlin JA. Does masking author identity improve peer review quality?. JAMA 1998;280(3):240‐2. - PubMed
McNutt 1990 {published data only}
    1. McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review. JAMA 1990;263(10):1371‐6. - PubMed
Neuhauser 1989 {published data only}
    1. Neuhauser D, Koran CJ. Calling Medical Care reviewers first: a randomized trial. Medical Care 1989;27(6):664‐6. - PubMed
Pierie 1996 {published data only}
    1. Pierie J, Walvoort HC, Overbeke AJPM. Readers' evaluation of effect of peer review and editing on quality of articles in the Nederlands Tijdschrift voor Geneeskunde. Lancet 1996;348:1480‐3. - PubMed
Pitkin 2002 {published data only}
    1. Pitkin RM, Burmeister LF. Identifying manuscript reviewers: randomized comparison of asking first or just sending. Journal of the American Medical Association 2002;287(21):2795‐2796. - PubMed
Resch 2000 {published data only}
    1. Resch KI, Ernst E, Garrow J. A randomized controlled study of reviewer bias against an unconventional therapy. Journal of the Royal Society of Medicine 2000;93(4.):164‐167. - PMC - PubMed
Rochon 2002 {published data only}
    1. Rochon PA, Bero LA, Bay AM, Gold JL, Dergal JM, Binns MA, Streiner DL, Gurwitz JH. Comparison of review articles published in peer‐reviewed and throwaway journals. Journal of the American Medical Association 2002;287(21):2853‐2856. - PubMed
Schroter 2004 {published data only}
    1. Schroter S, Black N, Evans S, Carpenter J, Godlee F, Smith R. Effects of training on quality of peer review: Randomised controlled trial. British Medical Journal. 2004;328(7441):673‐675. - PMC - PubMed
Strayhorn 1993 {published data only}
    1. Strayhorn J, McDermott JF, Tanguay P. An intervention to improve the reliability of manuscript reviews for the Journal of the American Academy of Child and Adolescent Psychiatry. Am J Psychiatry 1993;150(6):947‐52. - PubMed
van Rooyen 1998 {published data only}
    1. Rooyen S, Godlee F, Evans S, Smith R, Black N. Effect of blinding and unmasking on the quality of peer review. JGIM 1999;14(10):622‐4. - PMC - PubMed
    1. Rooyen S, Godlee F, Evans S, Smith R, Black N. Effect of blinding and unmasking on the quality of peer review. JAMA 1998;280(3):234‐7. - PubMed
van Rooyen 1999 {published data only}
    1. Rooyen S, Black N, Goodlee F. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. J Clin Epidemiol 1999;52(7):625‐629. - PubMed
    1. Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial. BMJ 1999;318:23‐7. - PMC - PubMed
Walsh 2000 {published data only}
    1. Walsh E, Rooney M, Appleby L, Wilkinson G. Open peer review: a randomised controlled trial. British Journal of Psychiatry 2000;176:47‐51. - PubMed
Weller 1996 {published data only}
    1. Weller AC. Editorial peer review: A comparison of authors publishing in two groups of U.S. medical journals. Bulletin of the Medical Library Association 1996;84(3):359‐366. - PMC - PubMed

References to studies excluded from this review

Abby 1994 {published data only}
    1. Abby M, Massey MD, Galandiuk S, Polk HC. Peer review is an effective screening process to evaluate medical manuscripts. JAMA 1994;272(2):105‐6. - PubMed
Bacchetti 2002 {published data only}
    1. Bacchetti P. Peer review of statistics in medical research: The other problem. British Medical Journal 2002;324(7348):1271‐1273. - PMC - PubMed
Baxt 1998 {published data only}
    1. Baxt WG, Waeckerle JF, Berlin JA, Callaham ML. Who reviewers the reviewers? Feasibility of using a fictitious manuscript to evaluate peer reviewer performance. Annals of Emergency Medicine 1998;32:310‐17. - PubMed
Blank 1991 {published data only}
    1. Blank RM. The effects of double‐blind versus single‐blind reviewing; experimental evidence from the American Economic Review. American Economic Review 1991;81(5):1041‐67.
Cho 1998 {published data only}
    1. Cho MK, Justice AC, Winker MA, Berlin JA, Waeckerle, Callaham ML, Rennie D. Masking author identity in peer review: what factors influence masking success?. JAMA 1998;280(3):243‐5. - PubMed
Cicchetti 1992 {published data only}
    1. Cicchetti DV, Rourke BP, Wass P. Peer review for manuscript and grant submissions: Relevance for research in clinical neuropsychology. Journal of Clinical and Experimental Neuropsychology 1992;14(6):976‐80. - PubMed
Cicchetti 1998 {published data only}
    1. Cicchetti DV. Good science and good peer reviewing: Are they related?. Journal of Clinical and Experimental Neuropsychology 1998;20(3):428‐31. - PubMed
Cleary 1988 {published data only}
    1. Cleary J, Alexander B. Blind versus non blind review: a reevaluation of selected medical journals. DICP, The Annals of Pharmacotherapy 1990;24:1117‐8. - PubMed
    1. Cleary JD, Alexander B. Blind versus nonblind review: Survey of selected medical journals. Drug Intelligance and Clinical Pharmacy 1988;22:601‐2. - PubMed
Coronel 1999 {published data only}
    1. Coronel R, Opthof T. The role of the reviewer in editorial decision‐making. Cardiovascular Research 1999;43:261‐4.
Cox 1993 {published data only}
    1. Cox D, Gleser L, Perlman M, Reid N, Roeder K. Report of the ad hoc committee on double‐blind refereeing. Statistical Science 1993;8(3):310‐30.
Cullen 1992 {published data only}
    1. Cullen DJ, Macaulay A. Consistency between peer reviewers for a clinical specialty journal. Academic Medicine 1992;67(12):856‐9. - PubMed
Dixon 1983 {published data only}
    1. Dixon GF, Schonfeld SA, Altman M, Whitcomb ME. The peer review and editorial process: A limited evaluation. The American Journal of Medicine 1983;74:494‐5. - PubMed
Feurer 1994 {published data only}
    1. Feurer ID, Becker GJ, Picus D, Ramirez E, Darcy MD, Hicks ME. Evaluating peer reviews: Pilot testing of a grading instrument. JAMA 1994;272(2):98‐100. - PubMed
Goldbeck‐Wood 1999 {published data only}
    1. Goldbeck‐Wood S. Secrecy and openness in peer review ‐ time for a change of culture?. Italian Journal of Gastroenterology and Hepatology 1999;31(8):659‐62. - PubMed
Hatch 1998 {published data only}
    1. Hatch CL, Goodman SN. Perceived value of providing peer reviewers with abstracts and preprints of related published and unpublished papers. JAMA 1998;280(3):273‐4. - PubMed
Hemlin 1999 {published data only}
    1. Hemlin S. (Dis) Agreement in peer review. In: Juslin P Montgomery H editor(s). Judgment and decision making: Neo‐Brunsurikian and process‐tracing approaches. Nahwah, New Jersey: Lawrance Erlbaum Associates, 1999:275‐301.
Jadad 1998 {published data only}
    1. Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, Moher M, Moher D. Methodology and reports of systematic reviews and meta‐analyses: A comparison of Cochrane reviews with articles published in paper‐based journals. JAMA 1998;280(3):278‐80. - PubMed
Katz 2002 {published data only}
    1. Katz DS, Proto AV, Olmsted WW. Incidence and nature of unblinding by authors: Our experience at two radiology journals with double‐blinded peer review policies. American Journal of Roentgenology 2002;179(6):1415‐1417. - PubMed
Kumar 1999 {published data only}
    1. Kumar PD. How do peer reviewers of journal articles perform? Evaluating the reviewers with a sham paper. J Assoc Physicians India 1999;47:198‐200. - PubMed
Laband 1994 {published data only}
    1. Laband DN, Piette MJ. Does the 'blindness' of peer review influence manuscript selection efficiency?. Southern Economic Journal 1994;60:896‐906.
Lee 2002 {published data only}
    1. Lee KP, Schotland M, Bacchetti P, Bero LA. Association of journal quality indicators with methodological quality of clinical research articles. Journal of the American Medical Association 2002;287(21):2805‐2808. - PubMed
Mahoney 1977 {published data only}
    1. Mahoney MJ. Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research 1977;1(2):161‐175.
Morrow 1992 {published data only}
    1. Morrow J, Bray M, Fulton J, Thomas J. Interrater reliability of 1987‐1991 Research Quarterly for Exercise and Sport reviews. Research Quarterly for Exercise and Sport 1992;63:200‐4. - PubMed
Nylenna 1994 {published data only}
    1. Nylenna M, Riis P, Karlsson Y. Multiple blinded reviews of the same two manuscripts: Effects of referee characteristics and publication language. JAMA 1994;272(2):149‐51. - PubMed
Opthof 1999 {published data only}
    1. Opthof T. Submission, acceptance rate, rapid review system and impact factor. Cardiovascular Research 1999;41:1‐4.
Oxman 1991 {published data only}
    1. Oxman AD, Guyatt GH, Singer J, Goldsmith CH, Hutchinson BG, Milner RA, Streiner DL. Agreement among reviewers of review articles. Journal of Clincal Epidemiology 1991;44(1):91‐8. - PubMed
Peters 1982 {published data only}
    1. Peters DP, Ceci SJ. Peer‐review practices of psychological journals: The fate of published articles, submitted again. The Behavioural and Brain Sciences 1982;5:187‐255.
Pitkin 2002a {published data only}
    1. Pitkin RM, Burmeister LF. Prodding tardy reviewers. A randomised comparison of telephone, fax, and e‐mail. JAMA 2002;287(21):2794‐5. - PubMed
Presser 1980 {published data only}
    1. Presser S. Collaboration and the quality of research. Social Studies of Science (SAGE, London & Beverly Hills) 1980;10:95‐101.
Purcell 1998 {published data only}
    1. Purcell GP, Donovan SL, Davidoff F. Changes to manuscripts during the editorial process: Characterizing the evolution of a clinical paper. JAMA 1998;280(3):227‐8. - PubMed
Rosenblatt 1980 {published data only}
    1. Rosenblatt A, Kirk SA. Recognition of authors in blind review of manuscripts. Journal of Social Service Research 1980;3(4):383‐94.
Schriger 2002 {published data only}
    1. Schriger DL, Cooper RJ, Wears RL, Waeckerle JF. The effect of dedicated methodology and statistical review on published manuscript quality. Annals of Emergency Medicine Schriger DL, Cooper RJ, Wears RL2002;40(3):334‐7. - PubMed
van Rooyen 1999b {published data only}
    1. van Rooyen, Black N, Godlee F. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. Journal of Clinical Epidemiology 1999;52:625‐9. - PubMed

References to ongoing studies

Delamothe/BMJ {published data only}
    1. Ongoing study Starting date of trial not provided. Contact author for more information.
Lee/Bero 2004 {published and unpublished data}
    1. A Qualitative Study of Editorial Decision Making. Ongoing study Starting date of trial not provided. Contact author for more information.
Schroter/BMJ 2004 {published data only}
    1. Ongoing study Starting date of trial not provided. Contact author for more information.

Additional references

Antman 1992
    1. Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. A comparison of results of meta‐analyses of randomized controlled trials and recommendations of clinical experts. JAMA 1992;268:240‐8. - PubMed
Godlee 2003
    1. Godlee F, Jefferson T. Introduction. In: Godlee F, Jefferson T editor(s). Peer review in health sciences. Second Edition. London: BMJ Books, 2003:xiii‐xv.
Hagstrom 1965
    1. Hagstrom WO. The Scientific Community. Southern Illinois University Press, 1965.
Horton 2000
    1. Horton R. The refiguration of medical thought. The Lancet 2000;356:2‐4. - PubMed
Jefferson 1999
    1. Jefferson T. What are the costs and benefits of editorials and non‐systematic reviews?. BMJ 1999;318:135. - PMC - PubMed
Jefferson 2002
    1. Jefferson TO, Alderson P, Davidoff F, Wager E. Effects of editorial peer review: a systematic review. JAMA 2002;287:2784‐2786. - PubMed
Jefferson 2002a
    1. Jefferson TO, Wager E, Davidoff F. Measuring the quality of editorial peer review. JAMA 2002;287:2786‐90. - PubMed
Knoll 1990
    1. Knoll E. The communities of scientists and journal peer‐review. JAMA 1990;263:1330‐2. - PubMed
Kronick 1990
    1. Kronick DA. Peer‐review in 18th‐century scientific journalism. JAMA 1990;263:1321‐2. - PubMed
Overbeke 2003
    1. Overbeke J, Wager E. The state of the evidence: what we know and what we don't know about journal peer review. In: Godlee F, Jefferson T editor(s). Peer review in health sciences. Second Edition. London: BMJ Books, 2003:45‐61.
Phillips 2000
    1. Phillips M. Peer review. The Lancet 2000;355:660. - PubMed
Rennie 1990
    1. Rennie D. Editorial peer‐review in biomedical publication. The first international congress. JAMA 1990;263:1317. - PubMed
Rennie 2003
    1. Rennie D. Editorial peer review:its development and rationale. In: Godlee F, Jefferson T editor(s). Peer review in health sciences. Second Edition. BMJ Books, 2003:1‐13.
Sackett 1997
    1. Sackett DL, Richardson WS, Rosenberg W, Haynes RB. Evidence‐Based Medicine. Edinburgh: Churchill Livingstone, 1997:8‐9.
Saint 2000
    1. Saint S, Christakis DA, Saha S, Elmore JG, Welsh DE, Baker P, et al. Journal reading habits of internists. J Gen Intern Med 2000;15:881‐4. - PMC - PubMed
Sievert 1996
    1. Sievert M, McKinin EJ, Johnson ED, Reid JC, Mitchell JA. Beyond relevance ‐ characteristics of key papers for clinicians: an exploratory study in an academic setting. Bull Med Libr Assoc 1996;84:351‐8. - PMC - PubMed
Vandenbroucke 1998
    1. Vandenbroucke JP. Medical journals and the shaping of medical knowledge. Lancet 1998;352:2001‐6. - PubMed
Wager 2007
    1. Wager E, Middleton P. Technical editing of research reports in biomedical journals. Cochrane Database of Systematic Reviews 2007, Issue 2. [DOI: 10.1002/14651858.MR000002.pub2] - DOI - PubMed

Publication types

MeSH terms

LinkOut - more resources