Assessment of potential bias in research grant peer review in Canada

Abstract

Background: Peer review is used to determine what research is funded and published, yet little is known about its effectiveness, and it is suspected that there may be biases. We investigated the variability of peer review and factors influencing ratings of grant applications.

Methods: We evaluated all grant applications submitted to the Canadian Institutes of Health Research between 2012 and 2014. The contribution of application, principal applicant and reviewer characteristics to overall application score was assessed after adjusting for the applicant's scientific productivity.

Results: Among 11 624 applications, 66.2% of principal applicants were male and 64.1% were in a basic science domain. We found a significant nonlinear association between scientific productivity and final application score that differed by applicant gender and scientific domain, with higher scores associated with past funding success and h-index and lower scores associated with female applicants and those in the applied sciences. Significantly lower application scores were also associated with applicants who were older, evaluated by female reviewers only (v. male reviewers only, -0.05 points, 95% confidence interval [CI] -0.08 to -0.02) or reviewers in scientific domains different from the applicant's (-0.07 points, 95% CI -0.11 to -0.03). Significantly higher application scores were also associated with reviewer agreement in application score (0.23 points, 95% CI 0.20 to 0.26), the existence of reviewer conflicts (0.09 points, 95% CI 0.07 to 0.11), larger budget requests (0.01 points per $100 000, 95% CI 0.007 to 0.02), and resubmissions (0.15 points, 95% CI 0.14 to 0.17). In addition, reviewers with high expertise were more likely than those with less expertise to provide higher scores to applicants with higher past success rates (0.18 points, 95% CI 0.08 to 0.28).

Interpretation: There is evidence of bias in peer review of operating grants that is of sufficient magnitude to change application scores from fundable to nonfundable. This should be addressed by training and policy changes in research funding.

Figure 1:

Fitted final scores of male and female applications in the domains of basic science (BS) and applied science (AS) in relation to past funding success rate based on the final model (Table 4). These graphs were generated using the reference category of all categorical variables in the final model, and the mean value for the continuous variables (mean number of investigators = 2, mean total amount of funding requested = $750 000, mean number of funded applications for the reviewers = 30%). For each of the subgroups, the h-index value was taken as the mean value within the specific group: biomedical sciences male = 7.9, biomedical sciences female = 6.3, applied sciences male = 7.9, applied sciences female = 5.5.