Publication Speed, Reporting Metrics, and Citation Impact of Cardiovascular Trials Supported by the National Heart, Lung, and Blood Institute

Abstract

Background: We previously demonstrated that cardiovascular (CV) trials funded by the National Heart, Lung, and Blood Institute (NHLBI) were more likely to be published in a timely manner and receive high raw citation counts if they focused on clinical endpoints. We did not examine the metrics of trial reports, and our citation measures were limited by failure to account for topic-related citation behaviors.

Methods and results: Of 244 CV trials completed between 2000 and 2011, we identified 184 whose main results were published by August 20, 2014. One investigator who was blinded to rapidity of publication and citation data read each publication and characterized it according to modified Delphi criteria. There were 46 trials (25%) that had Delphi scores of 8 or 9 (of a possible 9); these trials published faster (median time from trial completion to publication, 12.6 [interquartile range {IQR}, 6.7 to 23.3] vs. 21.8 [IQR, 12.1 to 34.9] months; P<0.01). They also had better normalized citation impact (median citation percentile for topic and date of publication, with 0 best and 100 worst, 1.92 [IQR, 0.64 to 7.83] vs. 8.41 [IQR, 1.80 to 24.75]; P=0.002). By random forest regression, we found that the 3 most important predictors of normalized citation percentile values were total costs, intention-to-treat analyses (as a modified Delphi quality measure), and focus on clinical (not surrogate) endpoints.

Conclusions: NHLBI CV trials were more likely to publish results quickly and yield higher topic-normalized citation impact if they reported results according to well-defined metrics, along with focus on clinical endpoints.

Keywords: bibliometrics; citation; public policy; randomized, controlled trial; research funding.

Figure 1

Kaplan–Meier plot showing cumulative publication rate according to modified Delphi reporting score. Of a possible score of 9, there were 46 trials that had a score of 8 or 9.

Figure 2

Box plot showing distribution of citation percentile according to modified Delphi score. Lower values are better and reflect higher citation rates normalized for topic and year of publication. Note the skew toward poorly cited articles (reflected by the mean values exceeding the median values).

Figure 3

Results of random forest regression model. Random forest variable importance measures (x-axis) are related to what degree of prediction can be explained by individual variables, after accounting for all other variables. The 3 most important correlates of normalized citation impact (“citation percentile”) were total costs, reporting of intention-to-treat analyses, and use of a clinical (as opposed to surrogate) primary endpoint.

Figure 4

Box plot showing associations of normalized citation impact (“citation percentile”) with use of clinical endpoints and costs. Random forests regression discovered an important interaction whereby citation percentile values were excellent (ie, low, given that lower values are better) if trials focused on clinical endpoints or if the budget exceeded $5 million; citation percentile values were substantially worse (higher) if neither characteristic was present. Of 184 projects, 118 (64%) focused on surrogate endpoints and had lower budgets, 37 (20%) focused on clinical endpoints and had higher budgets, and 29 (16%) had 1 (but not both) characteristic.

Figure 5

Results of secondary random forest regression model that included Journal Impact Factor as a predictor. Random forest variable importance measures (x-axis) are related to what degree of prediction can be explained by individual variables, after accounting for all other variables. The 4 most important correlates of normalized citation impact (“citation percentile”) were total costs, Journal Impact Factor, reporting of intention-to-treat analyses, and use of a clinical (as opposed to surrogate) primary endpoint.

Figure 6

Association of the citation percentile of main results articles and journal impact factor according to the random forest regression model depicted in Figure 5. The points and LOWESS (locally weighted scatterplot smoothing) smoother adjust for all other confounders. Lower citation percentiles imply greater citation impact; note that citation impact improves up to a journal impact factor of ≈15 to 20; for higher journal impact factors, citation impact remains consistently excellent (with all values at <10th percentile).