Integrative approach to quality assessment of medical journals using impact factor, eigenfactor, and article influence scores

Abstract

Background: Impact factor (IF) is a commonly used surrogate for assessing the scientific quality of journals and articles. There is growing discontent in the medical community with the use of this quality assessment tool because of its many inherent limitations. To help address such concerns, Eigenfactor (ES) and Article Influence scores (AIS) have been devised to assess scientific impact of journals. The principal aim was to compare the temporal trends in IF, ES, and AIS on the rank order of leading medical journals over time.

Methods: The 2001 to 2008 IF, ES, AIS, and number of citable items (CI) of 35 leading medical journals were collected from the Institute of Scientific Information (ISI) and the http://www.eigenfactor.org databases. The journals were ranked based on the published 2008 ES, AIS, and IF scores. Temporal score trends and variations were analyzed.

Results: In general, the AIS and IF values provided similar rank orders. Using ES values resulted in large changes in the rank orders with higher ranking being assigned to journals that publish a large volume of articles. Since 2001, the IF and AIS of most journals increased significantly; however the ES increased in only 51% of the journals in the analysis. Conversely, 26% of journals experienced a downward trend in their ES, while the rest experienced no significant changes (23%). This discordance between temporal trends in IF and ES was largely driven by temporal changes in the number of CI published by the journals.

Conclusion: The rank order of medical journals changes depending on whether IF, AIS or ES is used. All of these metrics are sensitive to the number of citable items published by journals. Consumers should thus consider all of these metrics rather than just IF alone in assessing the influence and importance of medical journals in their respective disciplines.

Figure 1. The Relationship Between Impact Factor (IF) and Eigenvalue Score (ES) In 2008.

The area of the circles is proportional to the number of citable items published in 2008. The area of the dotted line is expanded in figure 1B. R² = 0.5721; p<0.0001.

Figure 2. The Relationship of Changes in Citable Items Between 2001 and 2008 to Changes in Eigenfactor Score and Impact Factor Between 2001 and 2008.

R² = 0.1957; p = 0.0099 for the relationship between changes in citable items and changes in Eigenfactor score and R² = 0.1216; p = 0.0505* for the relationship between changes in citable items and changes in the impact factor. *The New England Journal of Medicine was excluded from the regression analysis, as it was an extreme outlier.