Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021;126(1):641-682.
doi: 10.1007/s11192-020-03758-1. Epub 2020 Nov 3.

A comprehensive quality assessment framework for scientific events

Sahar Vahdati  1   2 , Said Fathalla  3   4 , Christoph Lange  5   6 , Andreas Behrend  7 , Aysegul Say  3 , Zeynep Say  3 , Sören Auer  8   9
Affiliations

A comprehensive quality assessment framework for scientific events

Sahar Vahdati et al. Scientometrics. 2021.

Abstract

Systematic assessment of scientific events has become increasingly important for research communities. A range of metrics (e.g., citations, h-index) have been developed by different research communities to make such assessments effectual. However, most of the metrics for assessing the quality of less formal publication venues and events have not yet deeply investigated. It is also rather challenging to develop respective metrics because each research community has its own formal and informal rules of communication and quality standards. In this article, we develop a comprehensive framework of assessment metrics for evaluating scientific events and involved stakeholders. The resulting quality metrics are determined with respect to three general categories-events, persons, and bibliometrics. Our assessment methodology is empirically applied to several series of computer science events, such as conferences and workshops, using publicly available data for determining quality metrics. We show that the metrics' values coincide with the intuitive agreement of the community on its "top conferences". Our results demonstrate that highly-ranked events share similar profiles, including the provision of outstanding reviews, visiting diverse locations, having reputed people involved, and renowned sponsors.

Keywords: Bibliometrics; Metadata analysis; Quality assessment; Recommendation; Scientific events.

PubMed Disclaimer

Figures

Fig. 1
Fig. 1
List of services ordered by the extent to which their data is reusable and consideration of quality-related metrics
Fig. 2
Fig. 2
Dependency between dimensions
Fig. 3
Fig. 3
Distribution of event series AAAI over time and variation of acceptance rate considering the number of submissions and acceptances
Fig. 4
Fig. 4
The relevance of the defined metrics to different stakeholders of the scholarly communication
Fig. 5
Fig. 5
Relation of event location and people organizing the event, looking at last 11 years of the WWW conference (TheWebConf)
Fig. 6
Fig. 6
Relation of event location and people organizing the event, looking at the last 11 years of the VLDB conference
Fig. 7
Fig. 7
JCDL and SIGMOD/PODS paper acceptance rate and number of submitted/accepted papers for the last 10 years
Fig. 8
Fig. 8
Distribution of sponsors’ contributions, and amount of financial support of SIGMOD/PODS in 2018
Fig. 9
Fig. 9
Keynote speakers’ background percentage distribution in 2018 for all conferences in Table 2, and VLDB conference keynote speakers background distribution for the last 10 years
Fig. 10
Fig. 10
Publications-citations numbers and bibliometric information of ISWC for the last 10 years
See this image and copyright information in PMC

References

    1. Bizer C, Cyganiak R. Quality-driven information filtering using the WIQA policy framework. Journal of Web Semantics. 2009;7(1):1–10. doi: 10.1016/j.websem.2008.02.005. - DOI
    1. Bornmann L, Wolf M, Daniel H-D. Closed versus open reviewing of journal manuscripts: How far do comments differ in language use? Scientometrics. 2012;91(3):843–856. doi: 10.1007/s11192-011-0569-5. - DOI
    1. Bowyer, K. W. (2012). Mentoring Advice on “Conferences Versus Journals” for CSE Faculty.
    1. Bryl, V., Birukou, A., Eckert, K., & Kessler, M. (2014). What’s in the proceedings? Combining publisher’s and researcher’s perspectives. In SePublica. CEUR Workshop Proceedings.
    1. Caires, L. (2015). Again, the role of conference papers in computer science and informatics. http://ctp.di.fct.unl.pt/~lcaires/papers/conferencesCS.v4.pdf.

LinkOut - more resources