Observer agreement on the morphology of porous cranial lesions: Results from a workshop at the 2019 meeting of the Paleopathology Association

Abstract

Objective: This paper presents the results of a 2019 Paleopathology Association workshop that tested observer agreement on porous cranial lesion morphology and presence using multiple sets of existing guidelines for data collection.

Materials: Sixteen conference attendees of varying osteological experience served as observers. Three crania were assigned to each of four published guidelines for identifying and categorizing lesion morphology, for a total of twelve well-preserved human crania from the National Museum of Natural History Biological Anthropology Collections.

Methods: Observers assessed each cranium macroscopically according to its assigned set of guidelines.

Results: Observer concordance was higher using scoring guidelines with higher-quality photographs, such as the 2019 guidelines from Rinaldo and colleagues.

Conclusions: Data collection guidelines with high-quality color photos may support greater reliability of researcher-generated data on macroscopic skeletal features.

Significance: The conclusions of any research study are only as reliable as the data on which they are based. This work highlights the need for ongoing practices of quality control in a field in which much data results from individual judgement calls.

Limitations: Observer concordance is not a measure of observer accuracy. Sample size is insufficient to draw broadly generalizable conclusions on the reliability of data collected using the guidelines tested, and conference environments are not a facsimile of research settings.

Suggestions for further research: Iterative testing of methodological consistency using larger sample sizes and more non-pathological crania is advised to identify the factors that influence observer discordance and to improve guidelines for qualitative assessments.

Keywords: Categorical data; Cribra orbitalia; Interobserver error; Interrater reliability; Porotic hyperostosis.