Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2007 Apr 18;2007(2):MR000003.
doi: 10.1002/14651858.MR000003.pub2.

Peer review for improving the quality of grant applications

V Demicheli  1 , C Di Pietrantonj
Affiliations

Peer review for improving the quality of grant applications

V Demicheli et al. Cochrane Database Syst Rev. .

Abstract

Background: Grant giving relies heavily on peer review for the assessment of the quality of proposals but the evidence of effects of these procedures is scarce.

Objectives: To estimate the effect of grant giving peer review processes on importance, relevance, usefulness, soundness of methods, soundness of ethics, completeness and accuracy of funded research.

Search strategy: Electronic database searches and citation searches; researchers in the field were contacted.

Selection criteria: Prospective or retrospective comparative studies with two or more comparison groups assessing different interventions or one intervention against doing nothing. Interventions may regard different ways of screening, assigning or masking submissions, different ways of eliciting opinions or different decision making procedures. Only original research proposals and quality outcome measures were considered.

Data collection and analysis: Studies were read, classified and described according to their design and study question. No quantitative analysis was performed.

Main results: Ten studies were included. Two studies assessed the effect of different ways of screening submissions, one study compared open versus blinded peer review and three studies assessed the effect of different decision making procedures. Four studies considered agreement of the results of peer review processes as the outcome measure. Screening procedures appear to have little effect on the result of the peer review process. Open peer reviewers behave differently from blinded ones. Studies on decision-making procedures gave conflicting results. Agreement among reviewers and between different ways of assigning proposals or eliciting opinions was usually high.

Authors' conclusions: There is little empirical evidence on the effects of grant giving peer review. No studies assessing the impact of peer review on the quality of funded research are presently available. Experimental studies assessing the effects of grant giving peer review on importance, relevance, usefulness, soundness of methods, soundness of ethics, completeness and accuracy of funded research are urgently needed. Practices aimed to control and evaluate the potentially negative effects of peer review should be implemented meanwhile.

PubMed Disclaimer

Conflict of interest statement

None known

Update of

  • doi: 10.1002/14651858.MR000003

References

References to studies included in this review

Cole 1981 {published data only}
    1. Cole S, Cole J, Simon G. Chance and consensus in peer review. Science 1981;214:881‐886. - PubMed
Das 1985 {published data only}
    1. Das NK, Froehlich LA. Quantitative evaluation of peer review of program project and center applications in allergy and immunology. J Clin Immunol 1985;5:220‐227. - PubMed
Green 1989 {published data only}
    1. Green JG, Calhoun F, Nierzwicki L, Brackett J, Meier P. Rating intervals: an experiment in peer review. FASEB J 1989;3:1987‐1992. - PubMed
Hartmann 1990 {published data only}
    1. Hartmann I, Neidhardt F. Peer review at the Deutsche Forschungsgemeinschaft. Peer review at the Deutsche Forschungsgemeinschaft. Scientometrics 1990;19:419‐425.
Hodgson 1995 {published data only}
    1. Hodgson C. Evaluation of cardiovascular grant‐in‐aid applications by peer review: influence of internal and external reviewers and committees. Can J Cardiol 1995;11:864‐868. - PubMed
Hodgson 1997 {published data only}
    1. Hodgson C. How reliable is peer review? A comparison of operating grant proposals simultaneously submitted to two similar peer review systems. J Clin Epidem 1997;50:1189‐1195. - PubMed
Lee 2000 {published data only}
    1. Lee M, Om K, Koh J. The bias of sighted reviewers in research proposal evaluation: A comparative analysis of blind and open review in Korea. Scientometrics 2000;48(1):99‐116.
Russel 1983 {published data only}
    1. Russell AS, Thorn BD, Grace M. Peer review: a simplified approach. JRheumatol 1983;10:479‐481. - PubMed
Vener 1993 {published data only}
    1. Vener KJ, Feuer EJ, Gorelic L. A statistical model validating triage for the peer review process: keeping the competitive applications in the review pipeline. FASEB Journal 1993;7:1312‐1319. - PubMed
Wiener 1977 {published data only}
    1. Weiner S, Urivetsky M, Bregman D, et al. Peer review: inter‐reviewer agreement during evaluation of research grant evaluations. Clin Res 1977;25:306‐311. - PubMed

References to studies excluded from this review

Abrams 1991 {published data only}
    1. Abrams P. The predictive ability of peer review of grant proposals‐ the case of ecology and the United States National Science Foundation. Soc Stud Sci 1991;21:111‐132.
Anonimous 1994 {published data only}
    1. Anonimous. Peer Review: Reforms Needed to Ensure Fairness in Federal Agency Grant Selection. United States General Accounting Office. Washington DC 1994.
Anonimous 1995 {published data only}
    1. Anonimous. Peer Review: An Assessment of Recent Developments. Royal Society. London 1995.
Anonimous 1997 {published data only}
    1. Anonimous. Give him a grant, he's one of us. Research Fortnight 1997:13‐15.
Bailar 1991 {published data only}
    1. Bailar J. Reliability, fairness, objectivity and other inappropriate goals in peer‐review Behav Brain Sci. Behav Brain Sci 1991;14:137.
Birkett 1994 {published data only}
    1. Birkett N. The review process for applied research grant proposals: suggestions for revision. Canadian Medical Association Journal 1994;150:1227‐1229. - PMC - PubMed
Chubin 1990 {published data only}
    1. Chubin D, Hackett E. Peerless Science: Peer Review and U.S. Science Policy. Albany: SUNY Press 1990.
Chubin 1994 {published data only}
    1. Chubin D. Grants peer‐review in theory and practice. Evaluation Review 1994;18:20‐30.
Cicchetti 1991 {published data only}
    1. Cicchetti D. The reliability of peer review for manuscript and grant submissions: a cross‐disciplinary investigation. Behav Brain Sci 1991;14:119‐186.
Claveria 2000 {published data only}
    1. Claveria LE, Guallar E, Camì J, Conde J, Pastor R, et al. Does peer review predict the performance of research projects in health sciences?. Scientometrics 2000;47(1):11‐23.
Cole 1992 {published data only}
    1. Cole S. Making Science: Between Nature and Society. Cambridge: Harvard University Press, 1992.
Cunnigham 1993 {published data only}
    1. Cunnigham BL, Landis GH. A study of the outcome of the American Society for Aesthetic Plastic Surgery research grant program. Plastic and Reconstructive Surgery 1993;92(7):1397‐401. - PubMed
Fliesler 1997 {published data only}
    1. Fliesler SJ. Rethinking grant peer review. Science 1997;275:1399. - PubMed
Friesen 1998 {published data only}
    1. Friesen H. Equal opportunities in Canada. Nature 1998;391(326). - PubMed
Fuhrer 1985 {published data only}
    1. Fuhrer MJ, Grabois M. Grant application and review procedures of the National Institute of Handicapped Research: survey of applicant and peer reviewer opinions. Arch Phys Med Rehabil 1985;66:318‐321. - PubMed
Glantz 1994 {published data only}
    1. Glantz SA, Bero LA. Inappropriate and appropriate selection of 'peers' in grant review. JAMA 1994;272:114‐116. - PubMed
Grant 1997 {published data only}
    1. Grant J, Burden S, Breen G. No evidence of sexism in peer review. Nature 1997;390:438. - PubMed
Horrobin 1996 {published data only}
    1. Horrobin DF. Peer review of grant applications: a harbinger for mediocrity in clinical research?. Lancet 1996;348:1293‐1295. - PubMed
Horton 1996 {published data only}
    1. Horton R. Luck, lotteries and loopholes of grant review. Lancet 1996;348:1255‐1256. - PubMed
Kruytbosch 1989 {published data only}
    1. Kruytbosch C. The role and effectiveness of peer review. In: Evered D, Harnett S editor(s). The Evaluation of Scientific Research. Chichester: John Wiley, 1989:69‐85.
Marsh 1999 {published data only}
    1. Marsh HW, Bazeley P. Multiple evaluation of grant proposals by independent assessors: confirmatory factor analysis evaluations of reliability, validity and structure. Multivariate Behavioural Research 1999;34(1):1‐30. - PubMed
McCullough 1989 {published data only}
    1. McCullough J. First comprehensive survey of NSF applicants focuses on their concerns about proposal review. Sci Technol Human Values 1989;14:78‐88.
McCullough 1994 {published data only}
    1. McCullough J. The role and influence of the us national science foundation's program officers in reviewing and awarding grants. Higher Education 1994;28:85‐94.
Moxham 1992 {published data only}
    1. Moxham H, Anderson J. Peer review; a view from the inside. Science and Technology Policy 1992:7‐15.
Narin 1989 {published data only}
    1. Narin F. The impact of different modes of research funding. In: Evered D, Harnett S editor(s). The Evaluation of Scientific Research. Chichester: John Wiley, 1989.
VandenBeemt 1997 {published data only}
    1. Beemt F. The right mix: review by peers as well as by highly qualified persons (non peers). In: Wood F editor(s). Peer Review Process: Australian Research Council Commissioned Report: No 54. 1997:153‐164.
Wenneras 1997 {published data only}
    1. Wenneras C, Wold A. Nepotism and sexism in peer‐review. Nature 1997;387:341‐343. - PubMed

Additional references

Clarke 2002
    1. Clarke M, Oxman AD, editors. Cochrane Reviewers' Handbook 4.1.5. The Cochrane Library Issue 2. Oxford: Update Software, 2002.
Khalid 2000
    1. Khalid SK, Ter Riet G, Popay J, et al. Stage II Conducting the review: Phase 5 Study quality assessment. In: Khan KS, Ter Riet G, Glanville J, et al. editor(s). Undertaking Systematic reviews of research on effectiveness. CRD's guidance for carrying out or commissioning reviews. 2nd Edition. York: NHS Centre for Reviews and Dissemination University of York, 2000.
Kostoff 1994
    1. Kostoff R N. Research impact assessment. Principles and application to proposed, ongoing and completed projects. Invest Radiol 1994;29:864‐869. - PubMed
Roy 1985
    1. Roy R. Fundoing science: the real defects of peer review and an alternative to it. Sci Technol Human Values 1985;10:73‐81.
Smith 1988
    1. Smith R. Problems with peer review and alternatives. British Medical Journal 1988;296:774‐777. - PMC - PubMed
Wells 2000
    1. Wells GA, Shea B, O'Connel D, Peterson, Welch V, Losos M, Tugwll P. The Newcaste‐Ottawa Scale (NOS) for assessing the quality of non randomized studies in metanalyses. www.lri.ca/programs/ceu/oxford.htm 2000.
Wenneras 1999
    1. Wenneras C, World A. Bias in peer review of research proposals. In: Godlee F, Jefferson T editor(s). Peer review in health science. London: BMJ Books, 1999:79‐89.
Wessely 1999
    1. Wessely S. Peer review of grant application: a systematic review. In: Godlee F, Jefferson T editor(s). Peer review in health science. London: BMJ Books, 1999:14‐31.

Publication types

MeSH terms

LinkOut - more resources