Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Oct 19:10:e71368.
doi: 10.7554/eLife.71368.

An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes

Richard K Nakamura  1 , Lee S Mann  1 , Mark D Lindner  2 , Jeremy Braithwaite  3 , Mei-Ching Chen  2 , Adrian Vancea  2 , Noni Byrnes  2 , Valerie Durrant  2 , Bruce Reed  2
Affiliations

An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes

Richard K Nakamura et al. Elife. .

Abstract

Background: Blinding reviewers to applicant identity has been proposed to reduce bias in peer review.

Methods: This experimental test used 1200 NIH grant applications, 400 from Black investigators, 400 matched applications from White investigators, and 400 randomly selected applications from White investigators. Applications were reviewed by mail in standard and redacted formats.

Results: Redaction reduced, but did not eliminate, reviewers' ability to correctly guess features of identity. The primary, preregistered analysis hypothesized a differential effect of redaction according to investigator race in the matched applications. A set of secondary analyses (not preregistered) used the randomly selected applications from White scientists and tested the same interaction. Both analyses revealed similar effects: Standard format applications from White investigators scored better than those from Black investigators. Redaction cut the size of the difference by about half (e.g. from a Cohen's d of 0.20-0.10 in matched applications); redaction caused applications from White scientists to score worse but had no effect on scores for Black applications.

Conclusions: Grant-writing considerations and halo effects are discussed as competing explanations for this pattern. The findings support further evaluation of peer review models that diminish the influence of applicant identity.

Funding: Funding was provided by the NIH.

Keywords: cell biology; halo effects; medicine; none; peer review; racial bias; racial disparities; science funding.

PubMed Disclaimer

Conflict of interest statement

RN now retired, was Director of the NIH Center for Scientific Review (CSR) while the study was designed and implemented. LM now retired, was employed by CSR. ML, VD is employed by NIH/CSR, JB was employed by the contract research organization that conducted the data collection and initial analysis. MC MC is employed by NIH/CSR. AV is employed by NIH/Center for Scientific Review. NB is employed by NIH/Center for Scientific Review. She is the Director of CSR. BR is employed by NIH, he is the Deputy Director of CSR

Figures

Figure 1.
Figure 1.. Study background and timeline.
Figure 2.
Figure 2.. Distributions of preliminary overall impact scores according to race of PI and format in which the applications were reviewed.
Boxes delineate the central 50% of scores those falling between the 25th and 75th percentiles (Interquartile Range, IQR). Whiskers extend 1.5X the IQR. Dots mark outliers. Horizontal lines within boxes indicate the median, and “x” marks the mean value. Lower scores are better.
See this image and copyright information in PMC

Comment in

  • Blinding peer review.
    Taffe MA. Taffe MA. Elife. 2021 Nov 24;10:e74744. doi: 10.7554/eLife.74744. Elife. 2021. PMID: 34816796 Free PMC article.

References

    1. Aloisi A, Reid N. Un)Conscious Bias in the Astronomical Profession: Universal Recommendations to Improve Fairness, Inclusiveness, and Representation. arXiv. 2019 https://arxiv.org/abs/1907.05261
    1. Bendels MHK, Müller R, Brueggmann D, Groneberg DA. Gender disparities in high-quality research revealed by Nature Index journals. PLOS ONE. 2018;13:e0189136. doi: 10.1371/journal.pone.0189136. - DOI - PMC - PubMed
    1. Blank RM. The Effects of Double-Blind versus Single-Blind Reviewing: Experimental Evidence from The American Economic Review. The American Economic Review. 1991;81:1041–1067.
    1. Budden A, Tregenza T, Aarssen L, Koricheva J, Leimu R, Lortie C. Double-blind review favours increased representation of female authors. Trends in Ecology & Evolution. 2008;23:4–6. doi: 10.1016/j.tree.2007.07.008. - DOI - PubMed
    1. Campbell DT, Stanley JC. Experimental and Quasi-Experimental Designs for Research. Houghton Mifflin Company; 1963.

Publication types

MeSH terms