Do physicians judge a study by its cover? An investigation of journal attribution bias

Abstract

The effect of a journal's prestige on readers' impressions of an article is unknown. Two hypotheses were tested: first, that attribution of a study to a "high" prestige journal would be associated with improved impressions and attribution to a "low" prestige journal would be associated with diminished impressions; and second, that formal training in epidemiology and biostatistics would mitigate the effects of this journal attribution bias. The study was designed as a trial among a random sample of 264 internists. Participants were asked to read an article and an abstract from either the Southern Medical Journal (SMJ) or the New England Journal of Medicine (NEJM). Questionnaires were constructed that either attributed the article or abstract to its source or presented it as unattributed. After each article or abstract, respondents were asked to rate the quality of the study, the appropriateness of the methodology employed, the significance of the findings, and its likely effects on their practice. A 20-point impression score was created based on responses to these statements. The effect of attribution to a specific journal and formal epidemiology training on impression scores were assessed using linear regression. Of the 399 eligible participants, 264 questionnaires were returned (response rate 66%). Differences in impression scores associated with attribution of an article or abstract to the NEJM were.71 [95% C.I. (-.44-1.87)] and.50 [95% C.I. (-.87-1.87)] respectively; differences in impression scores associated with attribution of an article or abstract to the SMJ were -.12 [95% C.I. (-1.53-1.30)] and -.95 [95% C.I. (-2.41-.52)]. A stratified analysis demonstrated that epidemiology training did not meaningfully alter the effect of journal attribution on participants' impression scores. If journal attribution bias exists, it is likely to exert small and clinically insignificant effects when physicians read articles carefully. Formal training in epidemiology and biostatistics does not appear to alter these results.