Peer review at the American Journal of Roentgenology: how reviewer and manuscript characteristics affected editorial decisions on 196 major papers

Abstract

Objective: The objective of this study was to examine the relative influence of manuscript characteristics and peer-reviewer attributes in the assessment of manuscripts.

Materials and methods: Over a 6-month period, all major papers submitted to the American Journal of Roentgenology (AJR) were entered into a database that recorded manuscript characteristics, demographic profiles of reviewers, and the disposition of the manuscript. Manuscript characteristics included reviewer ratings on five scales (rhetoric, structure, science, import, and overall recommendation); the subspecialty class of the paper; the primary imaging technique; and the country of origin. Demographic profiles of the reviewers included age, sex, subspecialty, years of reviewing, academic rank, and practice type. Statistical analysis included correlation analysis, ordinal logistic regression, and analysis of variance.

Results: A total of 445 reviews of 196 manuscripts were the work of 335 reviewers. Of the 196 submitted manuscripts, 20 (10.2%) were accepted, 106 (54.1%) were rejected, and 70 (35.7%) were rejected with the opportunity to resubmit. Regarding manuscript characteristics, we found that the country of origin, score on the science scale, and score on the import scale were statistically significant variables for predicting the final disposition of a manuscript. Of the reviewer attributes, we found a statistically significant association between greater reviewer age and also higher academic rank with lower scores on the import scale. Reviewer concordance was higher for structure, science, and overall scores than on the rhetoric and import scores. Greater variability in the overall scoring of papers could be attributed to the reviewer than the manuscript, but both factors combined explain only 23% of the total variability.

Conclusion: At the AJR, manuscript acceptance was most strongly associated with reviewer scoring of the science and import of a major paper and also with the country of origin. Reviewers who were older and of higher academic rank tended to discount the importance of manuscripts.