Reviewing the reviewers: comparison of review quality and reviewer characteristics at the American Journal of Roentgenology

Abstract

Objective: The purpose of our study was to determine which manuscript reviewer characteristics are most strongly associated with reviewer performance as judged by editors of the American Journal of Roentgenology (AJR).

Materials and methods: At the AJR, manuscript reviews are rated by the journal editors on a subjective scale from 1 (lowest) to 4, on the basis of the value, thoroughness, and punctuality of the critique. We obtained all scores for AJR reviewers and determined the average score for each reviewer. We also sent a questionnaire to 989 reviewers requesting specific information regarding the age, sex, radiology subspecialty, number of years serving as a reviewer, academic rank, and practice type of the reviewer. The demographic profiles were correlated with the average quality score for each reviewer. Statistical analysis included correlation analysis and analysis of variance modeling. Reviewer quality scores were also correlated with the scoring of individual reviews and ultimate disposition of 196 manuscripts sent to the AJR during the same period.

Results: Responses to the questionnaire were obtained from 821 reviewers (83.0%), for whom quality scores were available for 714 (87.0%). Correlation analysis shows that the quality score of reviewers strongly correlated with younger age (p = 0.001). A statistically significant correlation between quality score and practice type was seen (p = 0.008), with reviewers from academic institutions receiving higher scores. No significant correlation was found between quality score and sex (p = 0.72), years of reviewing (p = 0.26), academic rank (p = 0.10), or the ultimate disposition of the manuscript (p = 0.40). The quality score of the reviewers showed no variation by subspecialty (p = 0.99).

Conclusion: The highest-rated AJR reviewers tended to be young and from academic institutions. The quality of peer review did not correlate with the sex, academic rank, or subspecialty of the reviewer.