Percentile ranking and citation impact of a large cohort of National Heart, Lung, and Blood Institute-funded cardiovascular R01 grants

Abstract

Rationale: Funding decisions for cardiovascular R01 grant applications at the National Heart, Lung, and Blood Institute (NHLBI) largely hinge on percentile rankings. It is not known whether this approach enables the highest impact science.

Objective: Our aim was to conduct an observational analysis of percentile rankings and bibliometric outcomes for a contemporary set of funded NHLBI cardiovascular R01 grants.

Methods and results: We identified 1492 investigator-initiated de novo R01 grant applications that were funded between 2001 and 2008 and followed their progress for linked publications and citations to those publications. Our coprimary end points were citations received per million dollars of funding, citations obtained <2 years of publication, and 2-year citations for each grant's maximally cited paper. In 7654 grant-years of funding that generated $3004 million of total National Institutes of Health awards, the portfolio yielded 16 793 publications that appeared between 2001 and 2012 (median per grant, 8; 25th and 75th percentiles, 4 and 14; range, 0-123), which received 2 224 255 citations (median per grant, 1048; 25th and 75th percentiles, 492 and 1932; range, 0-16 295). We found no association between percentile rankings and citation metrics; the absence of association persisted even after accounting for calendar time, grant duration, number of grants acknowledged per paper, number of authors per paper, early investigator status, human versus nonhuman focus, and institutional funding. An exploratory machine learning analysis suggested that grants with the best percentile rankings did yield more maximally cited papers.

Conclusions: In a large cohort of NHLBI-funded cardiovascular R01 grants, we were unable to find a monotonic association between better percentile ranking and higher scientific impact as assessed by citation metrics.

Keywords: National Heart, Lung, and Blood Institute (U.S.); bibliometrics.

Figure 1

Pareto plot of citations per grant for 1492 grants that met inclusion and exclusion criteria. The X-axis divides the grants into deciles according to the number of total citations that each grant’s papers received. The red bars shows the number of citations received within each decile of grants, while the line graph shows cumulative values going from the best to worst producing deciles of grants. The top 3 deciles (e.g. the 30% most productive grants) generated 76% of the total citations.

Figure 2

Bibliometric outcomes according to percentile ranking for 1492 R01 grants that met inclusion and exclusion criteria. All plots show values stratified by human or non-human study focus; curves were generated by LOESS fitting. Panel A shows data for citations received per $million allocated. Panel B shows data for 2-year citations. Panel C shows data for 2-year citations for each grant’s most highly cited paper (i.e., for each grant we identified which paper generated the most 2-year citations, and plot the number of citations generated by that paper according to percentile ranking). Panel D shows data for each grant’s h-index.

Figure 3

Citations per million dollars allocated according to percentile ranking stratified by the six study sections that reviewed the highest number of funded grants. Curves were generated by LOESS fitting.

Figure 4

Random forest findings. Panel A shows the relative importance of candidate variables for prediction of citations per $million allocated. The y-axis value corresponds to change in model discrimination by addition of that variable. Hence, mean number of grants acknowledged per paper is the strongest predictor, while percentile ranking is much weaker. Panel B shows the association between citations per $million (after logarithmic transformation) and percentile ranking after accounting for all other variables in the X axis of Panel A. Panel C shows corresponding values citations per $million and mean number of grants acknowledged per paper. Panel D shows in a different model the association between 2-year citations for each grant’s most highly cited paper and percentile ranking after accounting for all other variables in the X axis of Panel A plus average annual funding. In this model, the mean number of grants acknowledged per paper was again the strongest predictor, while percentile ranking was much weaker.

Figure 5

Bibliometric outcomes according to percentile ranking for 927 R01 grants that generated at least one paper that acknowledged only one grant. Panel A shows total citations. Panel B shows 2-year citations.