Prediction of junior faculty success in biomedical research: comparison of metrics and effects of mentoring programs

Abstract

Measuring and predicting the success of junior faculty is of considerable interest to faculty, academic institutions, funding agencies and faculty development and mentoring programs. Various metrics have been proposed to evaluate and predict research success and impact, such as the h-index, and modifications of this index, but they have not been evaluated and validated side-by-side in a rigorous empirical study. Our study provides a retrospective analysis of how well bibliographic metrics and formulas (numbers of total, first- and co-authored papers in the PubMed database, numbers of papers in high-impact journals) would have predicted the success of biomedical investigators (n = 40) affiliated with the University of Nevada, Reno, prior to, and after completion of significant mentoring and research support (through funded Centers of Biomedical Research Excellence, COBREs), or lack thereof (unfunded COBREs), in 2000-2014. The h-index and similar indices had little prognostic value. Publishing as mid- or even first author in only one high-impact journal was poorly correlated with future success. Remarkably, junior investigators with >6 first-author papers within 10 years were significantly (p < 0.0001) more likely (93%) to succeed than those with ≤6 first-author papers (4%), regardless of the journal's impact factor. The benefit of COBRE-support increased the success rate of junior faculty approximately 3-fold, from 15% to 47%. Our work defines a previously neglected set of metrics that predicted the success of junior faculty with high fidelity-thus defining the pool of faculty that will benefit the most from faculty development programs such as COBREs.

Keywords: Academic success; Bibliographic metrics; Biomedical research; Faculty development; Funding; Mentoring; Program evaluation; Research productivity; h-index.

Figure 1. Chances of junior faculty success plotted as a function of the h-index and the Ab index.

The chances of success of junior faculty at the University of Nevada, Reno from 2000 to 2014 are plotted as a function of the h-index (Hirsch, 2005) (in increments of 3–5, black font, lower x-axis) at the time of proposed start of COBRE mentoring. In addition, the chances of success are plotted as a function of the Ab-index (Biswal, 2013) that takes co-authorship into account (in increments of 50–100, grey, upper x-axis). The number of determinations for each data point is indicated (n). Neither of these indices predicted success with more than 60%, even for selected subgroups, and dropped off at higher values. Total n = 40, with n = 14 successful and n = 26 not successful.

Figure 2. Chances of junior faculty success plotted as a function of previously authored PubMed papers.

The chances of success of junior faculty at the University of Nevada, Reno from 2000 to 2014 are plotted as a function of the number of authored or co-authored papers in PubMed and published prior to the time of proposed start of COBRE mentoring (in increments of 5, black font, lower x-axis). In addition, the chances of success are plotted as a function of the number of 1st-author papers in journals with high impact factors (IF > 9) (in increments of 1, grey, upper x-axis). The number of determinations for each data point is indicated (n). Chances of success increased with larger numbers, but they reached 100% only with the highest values, limiting a meaningful prediction to a small percentage of faculty. Total n = 40, with n = 14 successful and n = 26 not successful.

Figure 3. Chances of junior faculty success as a function of previous 1st-authored papers in PubMed.

The chances of success of junior faculty at the University of Nevada, Reno from 2000 to 2014 are plotted as a function of the number of 1st-author papers listed in PubMed and published during the 10-year period prior to the date of the proposed start of COBRE mentoring (in increments of 1, x-axis). The success rate flips from 1/24 (4.1%) to 13/14 (92.9%) between six and seven 1st-authored papers in PubMed. Each data point represents between 1 and 7 faculty, total n = 40. The number of 1st-authored papers in the preceding decade predicted outcome in 38/40 (95%), and the difference of mean numbers of such papers per faculty, between successful and not successful faculty, was significant with p < 0.0001 (successful n = 14, not successful n = 26). The shaded area from five–nine 1st-authored papers defines the pool of junior faculty to benefit the most from a COBRE.