Journal impact factors do not equitably reflect academic staff performance in different medical subspecialties

Abstract

Background: The simplest variables to quantify on an academic curriculum vitae are the impact factors (IFs) of journals in which articles have been published. As a result, these measures are increasingly used as part of academic staff assessment. The present study tests the hypotheses that IFs exhibit patterns that are consistent between journals of different specialties and that these IFs reflect the quality of staff academic performance.

Methods: The IFs of a sample of journals from each of four medical specialties-medicine, oncology, genetics, and public and occupational health-were downloaded from the Science Citation Index and compared. Overall and specialty-specific journal IF frequencies were analyzed with respect to distribution patterns, averages, and skew.

Results: Approximately 91% of journal IFs fell within the 0 to 5 range, with 97% being less than 10. The overall IF distribution featured a positive skew and a mean of 2.5. Separate analysis of the journal specialty subsets revealed significant differences in IF means (genetics 3.4 > oncology 3.1 > medicine 2.0 > public health 1.6; p < .006), all of which well exceeded the respective IF medians. Journals from the general medicine category exhibited both the lowest IF median (0.7) and the most positively skewed distribution.

Conclusion: The distribution of IFs exhibits degrees of skew, numeric average, and spread that differ significantly between journal specialty subsets. This suggests that factors other than random variations underlie much of the IF variation between specialty journals and reduces the plausibility of a reliable correlation between IFs and the quality of academic staff performance. It is concluded that a dominant emphasis on IFs in academic recruitment and promotion may select for long-term faculty characteristics other than academic quality alone.