Metrics to evaluate implementation scientists in the USA: what matters most?

Abstract

Background: Implementation science has grown rapidly as a discipline over the past two decades. An examination of how publication patterns and other scholarly activities of implementation scientists are weighted in the tenure and promotion process is needed given the unique and applied focus of the field.

Methods: We surveyed implementation scientists (mostly from the USA) to understand their perspectives on the following matters: (1) factors weighted in tenure and promotion for implementation scientists, (2) how important these factors are for success as an implementation scientist, (3) how impact is defined for implementation scientists, (4) top journals in implementation science, and (5) how these journals are perceived with regard to their prestige. We calculated univariate descriptive statistics for all quantitative data, and we used Wilcoxon signed-rank tests to compare the participants' ratings of various factors. We analyzed open-ended qualitative responses using content analysis.

Results: One hundred thirty-two implementation scientists completed the survey (response rate = 28.9%). Four factors were rated as more important for tenure and promotion decisions: number of publications, quality of publication outlets, success in obtaining external funding, and record of excellence in teaching. Six factors were rated as more important for overall success as an implementation scientist: presentations at professional meetings, involvement in professional service, impact of the implementation scientist's scholarship on the local community and/or state, impact of the implementation scientist's scholarship on the research community, the number and quality of the implementation scientist's community partnerships, and the implementation scientist's ability to disseminate their work to non-research audiences. Participants most frequently defined and described impact as changing practice and/or policy. This expert cohort identified Implementation Science as the top journal in the field.

Conclusions: Overall, there was a significant mismatch between the factors experts identified as being important to academic success (e.g., tenure and promotion) and the factors needed to be a successful implementation scientist. Findings have important implications for capacity building, although they are largely reflective of the promotion and tenure process in the USA.

Keywords: Academic journals; Community partnerships; Faculty evaluation; Impact; Tenure and promotion.

Fig. 1

Additional factors reported as important for evaluating implementation scientists on their performance (n = 75)

Fig. 2

Coded definitions of impact of an implementation scientist’s work (n = 106)

Fig. 3

Coded descriptions of participants’ own work having an impact (n = 118)