Validity of the impact factor of journals as a measure of randomized controlled trial quality

Abstract

Objective: To assess whether the impact factor, a measure of the frequency with which journal articles are cited in the scientific literature, is a proxy measure of the quality of articles reporting the results of randomized controlled trials.

Method: The quality of trials included in an ongoing Cochrane review concerned with the antidepressant fluoxetine was assessed using the Cochrane Collaboration Depression, Anxiety, and Neurosis quality assessment instrument, the Jadad scale, and the quality criterion of the Cochrane Collaboration Handbook. Journal impact factors were extracted from the Journal Citation Report.

Results: A total of 131 articles reported results from 132 clinical trials comparing fluoxetine with other antidepressants. The relationship between trial quality and the impact factor of journals where these studies were published, stratified by period of publication, revealed that journals with impact factors above 4 points published only trials with above-average overall quality ratings, while journals with impact factors below 4 points published both high- and low-quality trials. The Jadad scale revealed similar quality in trials published in journals with high, medium, and low impact factors (Pearson chi(2) = 0.298, p = .861), and the quality criterion of the Cochrane Collaboration Handbook showed unclear randomization in the majority of trials and in all 15 trials published in high-impact factor journals (Pearson chi(2) = 4.678, p = .096).

Conclusion: The impact factor of journals is not a valid measure of randomized controlled trial quality.