Evaluation of cardiovascular grant-in-aid applications by peer review: influence of internal and external reviewers and committees

Abstract

Objective: To determine what factors of the peer review process played a significant role in the assignment of scores for scientific merit by a major cardiovascular research finding agency. Specific variables studied included scores of internal and external reviewers, committee assignment and year of application.

Design and setting: Retrospective analysis of research proposals submitted to the Heart and Stroke Foundation of Ontario from 1990 to 1994.

Subjects: Of a total database of 804 grant-in-aid (operating or project) proposals submitted to the Heart and Stroke Foundation of Ontario from 1990 to 1994, 779 had complete data and were included in the study. Variables analyzed were scores of internal and external reviewers, final committee score, committee assignment and year of application.

Main results: There was poor correlation (r = 0.113) between external reviewers of grant proposals and only slightly better correlation between internal reviewers (r = 0.331). When mean internal and external scores were analyzed, correlation improved (r = 0.451) but agreement beyond chance was only moderate (weighed Cohen's kappa = 0.532). Regression analysis established that mean internal score had a greater impact on final committee score (r2 = 0.607, P < 0.001) than mean internal score (r2 = 0.348, P < 0.001). When review committee was entered as a dummy variable into a regression statement, it was statistically significant (P < 0.001) but explained less than 8% of variance in the final committee score. Mean scores showed a small 'upward creep' over the five-year period but explained less than 4% of variance in final committee score (r2 = 0.039, P < 0.001).

Conclusions: In the peer review system studied, the scores of internal reviewers were more closely correlated to final committee score for scientific merit than those of external reviewers. Nevertheless, final committee scores were significantly different from either internal or external scores, suggesting that the process of committee discussion is an important, and highly influential, step in the peer review process.