Recruitment of reviewers is becoming harder at some journals: a test of the influence of reviewer fatigue at six journals in ecology and evolution

Abstract

Background: It is commonly reported by editors that it has become harder to recruit reviewers for peer review and that this is because individuals are being asked to review too often and are experiencing reviewer fatigue. However, evidence supporting these arguments is largely anecdotal.

Main body: We examine responses of individuals to review invitations for six journals in ecology and evolution. The proportion of invitations that lead to a submitted review has been decreasing steadily over 13 years (2003-2015) for four of the six journals examined, with a cumulative effect that has been quite substantial (average decline from 56% of review invitations generating a review in 2003 to just 37% in 2015). The likelihood that an invitee agrees to review declines significantly with the number of invitations they receive in a year. However, the average number of invitations being sent to prospective reviewers and the proportion of individuals being invited more than once per year has not changed much over these 13 years, despite substantial increases in the total number of review invitations being sent by these journals-the reviewer base has expanded concomitant with this growth in review requests.

Conclusions: The proportion of review invitations that lead to a review being submitted has been declining steadily for four of the six journals examined here, but reviewer fatigue is not likely the primary explanation for this decline.

Keywords: Peer review; Reviewer fatigue; Reviewers; Scholarly journals.

Fig. 1

Declines in reviewer recruitment success for standard research papers at six journals in the fields of ecology and evolution. The lines are the proportion of invited reviewers who responded to the invitation email (red line, filled circles), the proportion of respondees who agreed to review (blue line, triangles), the proportion of all invited reviewers who agreed to review (black line, open circles), and the proportion of all invitations that generated a submitted review (cyan line, squares). Standard papers include traditional research papers and excludes commentaries, perspectives, brief communications, and any other manuscript type not designated “original article” (Evolution), “research article” (Methods in Ecol Evol), or “standard paper” (the remaining journals). This also excludes revisions and, for Evolution, resubmissions of previously rejected papers. Analyses: Logistic regression, Response = Year + Journal + Year*Journal interaction, with Year as a continuous variable. (A) Proportion of invitees responding to invitation (red line, filled circles): Year: χ ² ₁ = 18.7, P < 0.001, Journal: χ ² ₁ = 109.2, P < 0.001, Interaction: χ ² ₁ = 109.2, P < 0.001; (B) proportion of respondees agreeing to review (blue line, triangles): Year: χ ² ₁ = 347.1, P < 0.001, Journal: χ ² ₁ = 150.7, P < 0.001, Interaction: χ ² ₁ = 151.1, P < 0.001; (C) proportion of invitees agreeing to review (black line, open circles), Year: χ ² ₁ = 352.6, P < 0.001, Journal: χ ² ₁ = 162.2, P < 0.001, Interaction: χ ² ₁ = 162.7, P < 0.001; (D) proportion of all invitations generating a review (cyan line, squares): Year: χ ² ₁ = 313.4, P < 0.001, Journal: χ ² ₁ = 171.0, P < 0.001, Interaction: χ ² ₁ = 171.5, P < 0.001

Fig. 2

The a total number of reviewers invited, b average number of invitations sent to each unique reviewer, and c proportion of reviewers invited more than one time within a given year, for “standard papers” (defined as in Fig. 1) submitted to six journals of ecology and evolution. The five journals published by the British Ecological Society (Functional Ecology, Journal of Animal Ecology, Journal of Applied Ecology, Journal of Ecology, and Methods in Ecology and Evolution) share a common reviewer database; the line labeled BES (top brown line, filled circles) is the average number of invitations sent to each unique reviewer across all five BES journals. These estimates (all lines in b and c) likely underestimate the true number of invitations sent to each researcher due to duplicate accounts in ScholarOne Manuscripts, though this error should be very small. We exclude 2015 (all journals), 2009 for Methods in Ecology and Evolution and 2007 for Evolution because we have data for only part of those year and thus numbers of invitations are not comparable with other years. Analysis for (B): Analysis of covariance, log(NumberOfTimesInvited) = Year + Journal + Year*Journal interaction, with year as a covariate; Year: F_1,81301 = 4.64, P = 0.03; Journal: F_5,81301 = 27.0, P < 0.001; Year*Journal: F_1,81301 = 27.1, P < 0.001. The means here differ slightly from those in Albert et al. (2016) for the journals / years in common between studies because (A) duplicate accounts are merged as found, reducing the number of unique reviewers and thus increasing our means per individual reviewer relative to theirs, and (B) in early years of the dataset their dataset double counts review invitations for reviewers of papers that were invited for revision, inflating their estimates for some journals and years (this is especially evident for Functional Ecology and Journal of Applied Ecology; see text for details)

Fig. 3

Reviewers who are invited more times within a single calendar year are more likely to decline to review. The Y-axis is the proportion of times an invitee agreed to review, averaged first across unique individuals within a calendar year and then across years within each journal. Invitations greater than 6 per year (x-axis) are excluded (treated as outliers) because sample sizes are very low and the patterns become uninterpretable. The figure and analysis excludes 2015 (all journals), 2009 for Methods in Ecology and Evolution and 2007 for Evolution because we have data for only part of those year and thus numbers of invitations are not comparable with other years. This analysis also excludes reviewers who did not respond to the email invitation; the overwhelming majority of non-responses were unique invitations, suggesting incorrect contact information. Analysis: Analysis of covariance, with each journal contributing one data point per reviewer invitation count per year; model: ProportionAgreed = Year + Journal + TimesInvited + 2-way interactions, with TimesInvited as a continuous variable; Year: F _11,253 = 1.13, P = 0.34; Journal: F _5,253 = 0.71, P = 0.62, TimesInvited: F _1,253 = 54.2, P < 0.001, Year*Journal: F _44,253 = 1.67, P = 0.008, Year*TimesInvited: F _11,253 = 4.64, P < 0.001, Journal*TimesInvited: F5,253 = 2.61, P = 0.03