And, not or: Quality, quantity in scientific publishing

Abstract

Scientists often perceive a trade-off between quantity and quality in scientific publishing: finite amounts of time and effort can be spent to produce few high-quality papers or subdivided to produce many papers of lower quality. Despite this perception, previous studies have indicated the opposite relationship, in which productivity (publishing more papers) is associated with increased paper quality (usually measured by citation accumulation). We examine this question in a novel way, comparing members of the National Academy of Sciences with themselves across years, and using a much larger dataset than previously analyzed. We find that a member's most highly cited paper in a given year has more citations in more productive years than in in less productive years. Their lowest cited paper each year, on the other hand, has fewer citations in more productive years. To disentangle the effect of the underlying distributions of citations and productivities, we repeat the analysis for hypothetical publication records generated by scrambling each author's citation counts among their publications. Surprisingly, these artificial histories re-create the above trends almost exactly. Put another way, the observed positive relationship between quantity and quality can be interpreted as a consequence of randomly drawing citation counts for each publication: more productive years yield higher-cited papers because they have more chances to draw a large value. This suggests that citation counts, and the rewards that have come to be associated with them, may be more stochastic than previously appreciated.

Fig 1. Sketch of the analysis.

A scientist’s publication record can be depicted by the number of papers published each year and the number of citations each paper has accrued. Our analysis is restricted to those papers published between 1980 and 2006. To evaluate whether or not a scientist experiences a trade-off between quantity (the number of papers published in a given year, indicated by the number of stacked pages) and quality (shown here as the number of citations accrued by those papers for simplicity, and indicated by the number within each page icon), we take each pair of papers and compare the number of papers published in the same year with the number of citations each paper has (A). If the lower cited paper comes from the year with fewer publications, then we call the pair concordant, else we call it discordant. we then look at the number of pairs falling into each category and calculate a correlation coefficient (Eq 1). To reduce potential biases introduced by considering all possible pairings, we can select a summary statistic and only consider pairs of this statistic between adjacent years, e.g. the maximally-cited paper in each year (B). To get an understanding of our expectations for the number of concordant versus discordant pairs, we can take an empirical time-line and randomize the citation counts among an author’s publications and re-run the analysis (C-D). Note that choosing the same pairs in a randomized timeline can result in the same or different relationships between the p’s and q’s. For each author, we are interested in the proportion of all possible pairings that are concordant. Proportions less than 0.5 correspond to a τ less than 0 and are indicative of a trade-off between quantity and quality.

Fig 2. Example trajectories of scientists who experience a trade-off or concordance.

The percent change in productivity (black) and quality (specifically the number of citations received by the maximally cited paper in that year; purple) from the previous year. Years in which the two changes are both of the same sign are termed concordant (green check-mark), while years in which the sign of the change differs are termed discordant (red X). The top scientist experiences a strong trade-off (low τ, many discordant years), while the bottom scientist experiences strong concordance between quantity and quality.

Fig 3. Violin plots of the strength of pairwise correlation between quantity and quality for members of the National Academy of Sciences.

The panels are divided based on which summary statistic is being compared across years (e.g. maximally-cited paper published in that year). A value of 1 (-1) indicates that, for every pair of adjacent years, the more productive one had a higher (lower) statistic. A value of 0 (horizontal black line) indicates that the larger statistic is equally likely to be from the more or less productive year. In blue (left in each plot) are the empirically observed correlation values for each author. In red (right in each plot) are the correlation values observed when citation counts were randomized within each author’s corpus.

Fig 4. The relationship between sample size and the maximum/minimum citation count drawn in that sample.

Right: histogram of citation counts for a given member of the National Academy of Sciences across their more than 1000 publications. Left: boxplots for the maximum (red, top) and minimum (orange, bottom) citation count drawn in samples of the size indicated by the horizontal axis. Each sample of a given size was repeated 1000 times to generate the distributions indicated by the boxplots.