'Your comments are meaner than your score': score calibration talk influences intra- and inter-panel variability during scientific grant peer review

doi:10.1093/reseval/rvw025

. 2017 Jan;26(1):1-14.

doi: 10.1093/reseval/rvw025. Epub 2017 Feb 14.

'Your comments are meaner than your score': score calibration talk influences intra- and inter-panel variability during scientific grant peer review

Elizabeth L Pier^{1

2}, Joshua Raclaw^{1

3}, Anna Kaatz¹, Markus Brauer⁴, Molly Carnes¹, Mitchell J Nathan², Cecilia E Ford^{1

5}

Affiliations

¹ Center for Women's Health Research, University of Wisconsin-Madison, 700 Regent Street, Ste. 301, Madison, WI 53715, USA.
² Department of Educational Psychology, University of Wisconsin-Madison, 1025 West Johnson Street, Madison, WI 53706, USA.
³ Department of English, West Chester University, 700 South High Street, West Chester, PA 19383, USA.
⁴ Department of Psychology, University of Wisconsin-Madison, 1202 West Johnson Street, Madison, WI 53706, USA.
⁵ Department of English, University of Wisconsin-Madison, 600 North Park Street, Madison, WI 53706, USA.

PMID: 28458466
PMCID: PMC5407376
DOI: 10.1093/reseval/rvw025

'Your comments are meaner than your score': score calibration talk influences intra- and inter-panel variability during scientific grant peer review

Elizabeth L Pier et al. Res Eval. 2017 Jan.

. 2017 Jan;26(1):1-14.

doi: 10.1093/reseval/rvw025. Epub 2017 Feb 14.

Authors

Elizabeth L Pier^{1

2}, Joshua Raclaw^{1

3}, Anna Kaatz¹, Markus Brauer⁴, Molly Carnes¹, Mitchell J Nathan², Cecilia E Ford^{1

5}

Affiliations

¹ Center for Women's Health Research, University of Wisconsin-Madison, 700 Regent Street, Ste. 301, Madison, WI 53715, USA.
² Department of Educational Psychology, University of Wisconsin-Madison, 1025 West Johnson Street, Madison, WI 53706, USA.
³ Department of English, West Chester University, 700 South High Street, West Chester, PA 19383, USA.
⁴ Department of Psychology, University of Wisconsin-Madison, 1202 West Johnson Street, Madison, WI 53706, USA.
⁵ Department of English, University of Wisconsin-Madison, 600 North Park Street, Madison, WI 53706, USA.

PMID: 28458466
PMCID: PMC5407376
DOI: 10.1093/reseval/rvw025

Abstract

In scientific grant peer review, groups of expert scientists meet to engage in the collaborative decision-making task of evaluating and scoring grant applications. Prior research on grant peer review has established that inter-reviewer reliability is typically poor. In the current study, experienced reviewers for the National Institutes of Health (NIH) were recruited to participate in one of four constructed peer review panel meetings. Each panel discussed and scored the same pool of recently reviewed NIH grant applications. We examined the degree of intra-panel variability in panels' scores of the applications before versus after collaborative discussion, and the degree of inter-panel variability. We also analyzed videotapes of reviewers' interactions for instances of one particular form of discourse-Score Calibration Talk-as one factor influencing the variability we observe. Results suggest that although reviewers within a single panel agree more following collaborative discussion, different panels agree less after discussion, and Score Calibration Talk plays a pivotal role in scoring variability during peer review. We discuss implications of this variability for the scientific peer review process.

Keywords: collaboration; decision making; discourse analysis; peer review.

PubMed Disclaimer

Conflict of interest statement

Conflict of interest statement. None declared.

Figures

**Figure 1**
Reverse nine-point scoring rubric used by NIH for the evaluation of R01 grants (adapted from NIH 2015).

**Figure 2**
Graph of the Krippendorff’s α statistics for the Preliminary Reviewer Scores, Final Reviewer Scores, and Final Panel Scores both within each CSS panel (‘intra-panel’) and between the four CSS panels (‘inter-panel’). A value of α ≥ 0.8 is considered ‘reliable’, and values between 0.67 and 0.8 are considered suitable for drawing ‘tentative conclusions’ (Krippendorff 2004). All applications are included.

**Figure 3**
Example #1 of S-SCT (from CSS1). The underlined portion of the transcript constitutes the S-SCT.

**Figure 4**
Example #2 of S-SCT (from CSS3). The underlined portion of the transcript constitutes the S-SCT.

**Figure 5**
Example #1 of O-SCT (from CSS2). Everything up until the Vice-chairperson begins a new project in line 39 is included in this instance of O-SCT (see Supplementary Appendix B).

**Figure 6**
Example #2 of O-SCT (from CSS1). The underlined portion of the transcript constitutes the O-SCT.

See this image and copyright information in PMC

Cited by

Community review: a robust and scalable selection system for resource allocation within open science and innovation communities.
Graham CLB, Landrain TE, Vjestica A, Masselot C, Lawton E, Blondel L, Haenal L, Greshake Tzovaras B, Santolini M. Graham CLB, et al. F1000Res. 2023 Apr 18;11:1440. doi: 10.12688/f1000research.125886.2. eCollection 2022. F1000Res. 2023. PMID: 38283124 Free PMC article.
The role of author identities in peer review.
Shah NB. Shah NB. PLoS One. 2023 Jun 21;18(6):e0286206. doi: 10.1371/journal.pone.0286206. eCollection 2023. PLoS One. 2023. PMID: 37342992 Free PMC article.
Biomedical research grant resubmission: rates and factors related to success - a scoping review.
Lasinsky AM, Wrightson J, Khan H, Moher D, Kitchin V, Khan K, Ardern CL. Lasinsky AM, et al. BMJ Open. 2024 Nov 14;14(11):e089927. doi: 10.1136/bmjopen-2024-089927. BMJ Open. 2024. PMID: 39542492 Free PMC article.
Laughter and the Chair: Social Pressures Influencing Scoring During Grant Peer Review Meetings.
Pier EL, Raclaw J, Carnes M, Ford CE, Kaatz A. Pier EL, et al. J Gen Intern Med. 2019 Apr;34(4):513-514. doi: 10.1007/s11606-018-4751-9. J Gen Intern Med. 2019. PMID: 30604119 Free PMC article. No abstract available.
Perspective commentary from the Society for Pediatric Research: supporting early-stage pediatric physician-scientist success.
Hurley E, Peeples ES, Bates CM, Hunstad DA, Barkin SL. Hurley E, et al. Pediatr Res. 2020 Apr;87(5):834-838. doi: 10.1038/s41390-019-0745-5. Epub 2020 Jan 2. Pediatr Res. 2020. PMID: 31896124 No abstract available.

See all "Cited by" articles

References

1. Aronson E, et al. The Jigsaw Classroom. Beverly Hills, CA: Sage Publishing Company; 1978.
1. Asch SE. Effects of Group Pressure on the Modification and Distortion of Judgments. In: Guetzkow H, editor. Groups, Leadership and Men. Pittsburgh, PA: Carnegie Press; 1951. pp. 177–90.
1. Asch SE. Opinions and Social Pressure. Scientific American. 1955;193:33–5.
1. Baker M, et al. The Role of Grounding in Collaborative Learning Tasks. In: Dillenbourg P, editor. Collaborative Learning: Cognitive and Computational Approaches. Oxford: Pergamon; 1999. pp. 31–63.
1. Barron B. Achieving Coordination in Collaborative Problem-Solving Groups. The Journal of the Learning Sciences. 2000;9(4):403–36.

Grants and funding

R01 GM111002/GM/NIGMS NIH HHS/United States

LinkOut - more resources

Full Text Sources
Other Literature Sources
- scite Smart Citations

[1] Aronson E, et al. The Jigsaw Classroom. Beverly Hills, CA: Sage Publishing Company; 1978.

[2] Aronson E, et al. The Jigsaw Classroom. Beverly Hills, CA: Sage Publishing Company; 1978.

[3] Asch SE. Effects of Group Pressure on the Modification and Distortion of Judgments. In: Guetzkow H, editor. Groups, Leadership and Men. Pittsburgh, PA: Carnegie Press; 1951. pp. 177–90.

[4] Asch SE. Effects of Group Pressure on the Modification and Distortion of Judgments. In: Guetzkow H, editor. Groups, Leadership and Men. Pittsburgh, PA: Carnegie Press; 1951. pp. 177–90.

[5] Asch SE. Opinions and Social Pressure. Scientific American. 1955;193:33–5.

[6] Asch SE. Opinions and Social Pressure. Scientific American. 1955;193:33–5.

[7] Baker M, et al. The Role of Grounding in Collaborative Learning Tasks. In: Dillenbourg P, editor. Collaborative Learning: Cognitive and Computational Approaches. Oxford: Pergamon; 1999. pp. 31–63.

[8] Baker M, et al. The Role of Grounding in Collaborative Learning Tasks. In: Dillenbourg P, editor. Collaborative Learning: Cognitive and Computational Approaches. Oxford: Pergamon; 1999. pp. 31–63.

[9] Barron B. Achieving Coordination in Collaborative Problem-Solving Groups. The Journal of the Learning Sciences. 2000;9(4):403–36.

[10] Barron B. Achieving Coordination in Collaborative Problem-Solving Groups. The Journal of the Learning Sciences. 2000;9(4):403–36.

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

'Your comments are meaner than your score': score calibration talk influences intra- and inter-panel variability during scientific grant peer review

Affiliations

'Your comments are meaner than your score': score calibration talk influences intra- and inter-panel variability during scientific grant peer review

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

Similar articles

Cited by

References

Grants and funding

LinkOut - more resources

Full Text Sources

Other Literature Sources