Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Comparative Study
. 2019 Sep 11;14(9):e0221212.
doi: 10.1371/journal.pone.0221212. eCollection 2019.

Citation gaming induced by bibliometric evaluation: A country-level comparative analysis

Alberto Baccini  1 , Giuseppe De Nicolao  2 , Eugenio Petrovich  1
Affiliations
Comparative Study

Citation gaming induced by bibliometric evaluation: A country-level comparative analysis

Alberto Baccini et al. PLoS One. .

Abstract

It is several years since national research evaluation systems around the globe started making use of quantitative indicators to measure the performance of researchers. Nevertheless, the effects on these systems on the behavior of the evaluated researchers are still largely unknown. For investigating this topic, we propose a new inwardness indicator able to gauge the degree of scientific self-referentiality of a country. Inwardness is defined as the proportion of citations coming from the country over the total number of citations gathered by the country. A comparative analysis of the trends for the G10 countries in the years 2000-2016 reveals a net increase of the Italian inwardness. Italy became, both globally and for a large majority of the research fields, the country with the highest inwardness and the lowest rate of international collaborations. The change in the Italian trend occurs in the years following the introduction in 2011 of national regulations in which key passages of professional careers are governed by bibliometric indicators. A most likely explanation of the peculiar Italian trend is a generalized strategic use of citations in the Italian scientific community, both in the form of strategic author self-citations and of citation clubs. We argue that the Italian case offers crucial insights on the constitutive effects of evaluation systems. As such, it could become a paradigmatic case in the debate about the use of indicators in science-policy contexts.

PubMed Disclaimer

Conflict of interest statement

The authors have declared that no competing interests exist.

Figures

Fig 1
Fig 1. Inwardness for G10 countries (2000-2016).
Source: elaboration on SCIval data.
Fig 2
Fig 2. Inwardness versus average international collaboration for the G10 countries.
The average international collaboration is the 3-year moving average calculated starting from the considered year. The international collaboration is defined as the share of publications of a country coauthored by at least a coauthor of a different country. Source: elaboration from SCIval data.
Fig 3
Fig 3. Inwardness delta in Scopus Main Categories in the periods 2000-2008 (A) and 2008-2016 (B).
ABS = Agricultural and Biological Sciences, AH = Arts and Humanities, BGMB = Biochemistry, Genetics and Molecular Biology, BMA = Business, Management and Accounting, CE = Chemical Engineering, CHEM = Chemistry, CS = Computer Science, DEN = Dentistry, DS = Decision Sciences, E = Energy, EEF = Economics, Econometrics and Finance, ENG = Engineering, EPS = Earth and Planetary Sciences, ES = Environmental Science, HP = Health Professions, IM = Immunology and Microbiology, MAT = Mathematics, MED = Medicine, MS = Materials Science, MUL = Multidisciplinary, NEU = Neuroscience, NUR = Nursing, PA = Physics and Astronomy, PSY = Psychology, PTP = Pharmacology, Toxicology and Pharmaceutics, SOC = Social Sciences, VET = Veterinary. Source: elaboration from SCIval data.
See this image and copyright information in PMC

Comment in

  • Citation doping not for Italy's elites.
    D'Antuono P, Ciavarella M. D'Antuono P, et al. Nature. 2019 Oct;574(7778):333. doi: 10.1038/d41586-019-03119-w. Nature. 2019. PMID: 31616097 No abstract available.

References

    1. Hicks D. Performance-based university research funding systems. Research Policy. 2012;41(2):251–261. 10.1016/j.respol.2011.09.007 - DOI
    1. Whitley R, Gläser J, editors. The changing governance of the sciences: the advent of research evaluation systems No. v. 26 in Sociology of the sciences yearbook. Dordrecht, the Netherlands: Springer; 2007. - PubMed
    1. Haustein S, Larivière V. The Use of Bibliometrics for Assessing Research: Possibilities, Limitations and Adverse Effects In: Welpe IM, Wollersheim J, Ringelhan S, Osterloh M, editors. Incentives and Performance. Cham: Springer International Publishing; 2015. p. 121–139. Available from: http://link.springer.com/10.1007/978-3-319-09785-5_8. - DOI
    1. Moed HF. Citation analysis in research evaluation No. v. 9 in Information science and knowledge management. Dordrecht: Springer; 2005.
    1. Geuna A, Martin BR. University Research Evaluation and Funding: An International Comparison. Minerva. 2003;41(4):277–304. 10.1023/B:MINE.0000005155.70870.bd - DOI

Publication types