Citation bias favoring statistically significant studies was present in medical research

Abstract

Objective: Statistically significant studies may be cited more than negative studies on the same topic. We aimed to assess here whether such citation bias is present across the medical literature.

Study design and setting: We conducted a cohort study of the association between statistical significance and citations. We selected all therapeutic intervention studies included in meta-analyses published between January and March 2010 in the Cochrane database, and retrieved citation counts of all individual studies using ISI Web of Knowledge. The association between the statistical significance of each study and the number of citations it received between 2008 and 2010 was assessed in mixed Poisson models.

Results: We identified 89 research questions addressed in 458 eligible articles. Significant studies were cited twice as often as nonsignificant studies (multiplicative effect of significance: 2.14, 95% confidence interval: 1.38-3.33). This association was partly because of the higher impact factor of journals where significant studies are published (adjusted multiplicative effect of significance: 1.14, 95% confidence interval: 0.87-1.51).

Conclusion: A citation bias favoring significant results occurs in medical research. As a consequence, treatments may seem more effective to the readers of medical literature than they really are.