Consistency between peer reviewers for a clinical specialty journal

Abstract

To analyze the consistency between independent peer reviewers in evaluating and ranking unsolicited articles, the authors used paired reviews of 422 unsolicited submissions to the Journal of Clinical Anesthesia from the end of 1988 through 1991. (The editors of this journal base their publication decisions, to a substantial degree, on congruence of their reviewers' recommendations). The reviewers were chosen for their interest in reviewing and areas of expertise. Their recommendations were ranged along a continuum of four categories: (1) accept outright, (2) accept with revision, (3) reject in present form (article could be revised and submitted again as a new submission), and (4) reject outright. The pairs of peer reviewers were consonant for 169 papers (40%), differed by one category for 168 papers (40%), differed by two categories for 73 papers (17%), and differed by three categories for 12 papers (3%). Thus, most articles' reviews were in consonance or close to it; articles reviewed by two members of the editorial board, however, were significantly less likely to be consonant (32%) than were those reviewed by two nonmembers (44%, chi-square, p = .027).