Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 May 14:9:366.
doi: 10.12688/f1000research.23418.2. eCollection 2020.

Use of the journal impact factor for assessing individual articles: Statistically flawed or not?

Ludo Waltman  1 , Vincent A Traag  1
Affiliations

Use of the journal impact factor for assessing individual articles: Statistically flawed or not?

Ludo Waltman et al. F1000Res. .

Abstract

Most scientometricians reject the use of the journal impact factor for assessing individual articles and their authors. The well-known San Francisco Declaration on Research Assessment also strongly objects against this way of using the impact factor. Arguments against the use of the impact factor at the level of individual articles are often based on statistical considerations. The skewness of journal citation distributions typically plays a central role in these arguments. We present a theoretical analysis of statistical arguments against the use of the impact factor at the level of individual articles. Our analysis shows that these arguments do not support the conclusion that the impact factor should not be used for assessing individual articles. Using computer simulations, we demonstrate that under certain conditions the number of citations an article has received is a more accurate indicator of the value of the article than the impact factor. However, under other conditions, the impact factor is a more accurate indicator. It is important to critically discuss the dominant role of the impact factor in research evaluations, but the discussion should not be based on misplaced statistical arguments. Instead, the primary focus should be on the socio-technical implications of the use of the impact factor.

Keywords: Journal impact factor; citation; research assessment; research evaluation; skewness.

PubMed Disclaimer

Conflict of interest statement

No competing interests were disclosed.

Figures

Figure 1.
Figure 1.. Accuracy of the impact factor (IF) and of citations for different values of σr2 and σc2.
Figure 2.
Figure 2.. Accuracy of the impact factor (IF) and of citations for different numbers of journals and for different values of σc2.
Figure 3.
Figure 3.. Accuracy of different hybrid indicators combining the impact factor (IF) and citations for different values of σc2.
See this image and copyright information in PMC

References

    1. Abramo G, D’Angelo CA, Di Costa F: Citations versus journal impact factor as proxy of quality: Could the latter ever be preferable? Scientometrics. 2010;84(3):821–833. 10.1007/s11192-010-0200-1 - DOI
    1. Allen L, Jones C, Dolby K, et al. : Looking for landmarks: the role of expert review and bibliometric analysis in evaluating scientific publication outputs. PLoS One. 2009;4(6):e5910. 10.1371/journal.pone.0005910 - DOI - PMC - PubMed
    1. Ancaiani A, Anfossi AF, Barbara A, et al. : Evaluating scientific research in Italy: The 2004-10 research evaluation exercise. Res Eval. 2015;24(3):242–255. 10.1093/reseval/rvv008 - DOI
    1. Anfossi A, Ciolfi A, Costa F, et al. : Large-scale assessment of research outputs through a weighted combination of bibliometric indicators. Scientometrics. 2016;107(2):671–683. 10.1007/s11192-016-1882-9 - DOI
    1. Barabasi AL, Albert R: Emergence of scaling in random networks. Science. 1999;286(5439):509–512. 10.1126/science.286.5439.509 - DOI - PubMed