Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2009 Jun 18;4(6):e5910.
doi: 10.1371/journal.pone.0005910.

Looking for landmarks: the role of expert review and bibliometric analysis in evaluating scientific publication outputs

Liz Allen  1 , Ceri Jones, Kevin Dolby, David Lynn, Mark Walport
Affiliations

Looking for landmarks: the role of expert review and bibliometric analysis in evaluating scientific publication outputs

Liz Allen et al. PLoS One. .

Abstract

Objective: To compare expert assessment with bibliometric indicators as tools to assess the quality and importance of scientific research papers.

Methods and materials: Shortly after their publication in 2005, the quality and importance of a cohort of nearly 700 Wellcome Trust (WT) associated research papers were assessed by expert reviewers; each paper was reviewed by two WT expert reviewers. After 3 years, we compared this initial assessment with other measures of paper impact.

Results: Shortly after publication, 62 (9%) of the 687 research papers were determined to describe at least a 'major addition to knowledge' -6 were thought to be 'landmark' papers. At an aggregate level, after 3 years, there was a strong positive association between expert assessment and impact as measured by number of citations and F1000 rating. However, there were some important exceptions indicating that bibliometric measures may not be sufficient in isolation as measures of research quality and importance, and especially not for assessing single papers or small groups of research publications.

Conclusion: When attempting to assess the quality and importance of research papers, we found that sole reliance on bibliometric indicators would have led us to miss papers containing important results as judged by expert review. In particular, some papers that were highly rated by experts were not highly cited during the first three years after publication. Tools that link expert peer reviews of research paper quality and importance to more quantitative indicators, such as citation analysis would be valuable additions to the field of research assessment and evaluation.

PubMed Disclaimer

Conflict of interest statement

Competing Interests: The authors have declared that no competing interests exist.

Figures

Figure 1
Figure 1. ‘PubMed 1000’ original research papers – ‘importance rating’.
Figure 2
Figure 2. ‘PubMed 1000’ – ‘importance rating’ & Journal Impact Factor (2005).
Figure 3
Figure 3. ‘PubMed 1000’ - ‘importance rating’ (2005) & citations (2008).
Figure 4
Figure 4. ‘PubMed 1000’ – ‘importance rating’ (2005) & F1000 rating (2008).
Figure 5
Figure 5. ‘PubMed 1000’ – number of authors & citations (2008).
See this image and copyright information in PMC

References

    1. Seglen PO. Why the impact factor of journals should not be used for evaluating research. BMJ. 1997;314:498–502. - PMC - PubMed
    1. Adam D. Citation analysis: The counting house. Nature. 2002;415:726–729. - PubMed
    1. Berghmans T, et al. Citation indexes do not reflect methodological quality in lung cancer randomised trials. Annals of Oncology. 2003;14:715–721. - PubMed
    1. Garfield E. The History and Meaning of the Journal Impact Factor. JAMA. 2006;295:90–93. - PubMed
    1. Ross JS. Effect of Blinded Peer Review on Abstract Acceptance. JAMA. 2006;295:1675–1680. - PubMed