Multiple blinded reviews of the same two manuscripts. Effects of referee characteristics and publication language

Abstract

Objective: To study the association between referee characteristics and their manuscript assessments, the influence of manuscript language on referees' judgments, and the usefulness, quality, and extent of referees' free-text comments.

Design: Two nonauthentic, but realistic, short manuscripts with a number of common methodological flaws were sent to 180 Scandinavian referees. Through randomization, each referee received one of the manuscripts in English and the other manuscript in the national language. A structured assessment of the manuscript quality was expressed on a 5-point scale, and the impact of referee characteristics (age, gender, experience, and so on) was analyzed by multiple linear regression.

Main outcome: Manuscript quality assessed by referees.

Results: A total of 312 reviews from 156 referees could be used for the study of referee characteristics and language. With increasing experience, the referees gave lower quality scores (P < .05). A tendency toward stricter assessment with younger age was seen (P < .05). No influence of referees' gender, specialty, or nationality was found. For the test manuscript of the poorest quality, the English version was assessed to be better than the national-language version (P < .05). A total of 159 of 312 reviews included free-text comments applicable for the methodological study. In 54 reviews (34%), no methodological comments accompanied the assessment, and in six reviews they were only incomplete. Wrong sampling unit was mentioned by one fourth of 80 referees. Only one referee mentioned the incorrect use of a parametric test in the analysis of data whose distribution was nonparametric.

Conclusions: Experienced and young referees gave a stricter assessment of the manuscripts than their less experienced and older colleagues. An English version seemed to be accepted more easily than a national-language version of the same manuscript. Most referees spontaneously mentioned the shortcomings of the manuscripts only as part of their overall judgment.