Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Review
. 2023;11(2):94-101.
doi: 10.22038/ABJS.2022.44387.2214.

Methodology for Online Reliability Studies: A Primer for Orthopedic Surgeons

Femke M A P Claessen  1 , Ruben Zwiers  2 , Thierry G Guitton  3 , Job N Doornberg  4
Affiliations
Review

Methodology for Online Reliability Studies: A Primer for Orthopedic Surgeons

Femke M A P Claessen et al. Arch Bone Jt Surg. 2023.

Abstract

In orthopedic surgery, there is an increasing number of papers about online studies on the reliability of classification systems. Useful classification systems need to be reliable and valid. Measurement of validity can be variable and is prone to observer bias. These online collaboratives derive adequate power to study reliability by having a large group of trained surgeons review a small number of cases instead of the "classic" reliability studies in which a small number of observers evaluate many cases. Large online interobserver studies have advantages (i.e., less than 15 minutes to complete the survey, the ability to randomize, and the ability to study factors associated with reliability, accuracy, or decision-making). This 'handbook' paper gives an overview of current methods for online reliability studies. We discuss the study design, sample size calculation, statistical analyses of results, pitfalls, and limitations of the study design.

Keywords: Interobserver studies; Methods; Reliability.

PubMed Disclaimer

Figures

Figure 1
Figure 1
Flowchart of the methodological process for reliability studies
See this image and copyright information in PMC

References

    1. Kocher MS, Zurakowski D. Clinical epidemiology and biostatistics: a primer for orthopaedic surgeons. J Bone Joint Surg Am. 2004;86-A(3):607–20. - PubMed
    1. Hegedus EJ, Stern B. Beyond SpPIN and SnNOUT: Considerations with Dichotomous Tests During Assessment of Diagnostic Accuracy. J Man Manip Ther. 2009;17(1):E1–5. - PMC - PubMed
    1. Doornberg J, Lindenhovius A, Kloen P, van Dijk CN, Zurakowski D, Ring D. Two and three-dimensional computed tomography for the classification and management of distal humeral fractures Evaluation of reliability and diagnostic accuracy. J Bone Joint Surg Am. 2006;88(8):1795–801. - PubMed
    1. Obuchowski NA, Gallas BD, Hillis SL. Multi-reader ROC studies with split-plot designs: a comparison of statistical methods. Acad Radiol. 2012;19(12):1508–17. - PMC - PubMed
    1. Lindenhovius A, Karanicolas PJ, Bhandari M, van Dijk N, Ring D. Collaboration for Outcome Assessment in Surgical T Interobserver reliability of coronoid fracture classification: two-dimensional versus three-dimensional computed tomography. J Hand Surg Am. 2009;34(9):1640–6. - PubMed

LinkOut - more resources