Citation counts and journal impact factors do not capture some indicators of research quality in the behavioural and brain sciences

Abstract

Citation data and journal impact factors are important components of faculty dossiers and figure prominently in both promotion decisions and assessments of a researcher's broader societal impact. Although these metrics play a large role in high-stakes decisions, the evidence is mixed about whether they are strongly correlated with indicators of research quality. We use data from a large-scale dataset comprising 45 144 journal articles with 667 208 statistical tests and data from 190 replication attempts to assess whether citation counts and impact factors predict three indicators of research quality: (i) the accuracy of statistical reporting, (ii) the evidential value of the reported data and (iii) the replicability of a given experimental result. Both citation counts and impact factors were weak and inconsistent predictors of research quality, so defined, and sometimes negatively related to quality. Our findings raise the possibility that citation data and impact factors may be of limited utility in evaluating scientists and their research. We discuss the implications of these findings in light of current incentive structures and discuss alternative approaches to evaluating research.

Keywords: bibliometrics; citation counts; open science; research quality.

Figure 1.

Number of statistical decision errors per 100 tests as a function of (a) number of times the article was cited, (b) journal impact factor, (c) number of authors on the paper and (d) year of publication. Shaded regions indicate 95% credible intervals. Analyses based on N = 45 144 published articles.

Figure 2.

Bayesian estimated posterior probability for H₁ for all t-tests and one degree of freedom F-tests that were statistically significant at the p < 0.05 level (N = 299 316) as a function of (a) number of times the article was cited, (b) 2017 journal impact factor, (c) year of publication, (d) number of authors on the paper and (e) reported degrees of freedom. Shaded regions indicate 95% credible intervals.

Figure 3.

Bayesian estimated posterior probability of replication success as a function of (a) 2017 journal impact factor, (b) number of times article was cited and (c) number of authors on the paper. Shaded regions indicate 95% credible intervals. Ticks on the x-axis indicate the distribution of individual cases for the predictor variable.

Figure 4.

Violin plots of (a) number of times each paper is cited, (b) number of times the first author was cited, (c) rated institutional prestige and (d) rated surprisingness of experimental findings split by studies that were and were not successfully replicated. Medians are given by boxed numbers; means are unboxed. Plots are based on the N = 100 replication attempts included in [58].