Assessing Binary Diagnoses of Bio-behavioral Disorders: The Clinical Relevance of Cohen's Kappa

Abstract

Cohen's (Educ Psychol Meth. 1960;23:37-40) kappa statistic has been criticized recently for providing low chance-corrected coefficients (<0.40) despite high levels of examiner agreement (e.g., ≥85%). The AC1 statistic (Series. 2002;2:1-9; BMC Med Res Methodol. 2013;13:61), the alpha statistic (J Clin Epidemiol. 1996;49:775-782), the B statistic (BMC Med Res Methodol. 2013;13:97), and the delta statistic (Commun Statist Theory Meth. 2008;37:760-772) were developed to "correct" this "problem." Each of these statistics produces inaccurate results because they either derive from levels of chance agreement that are incorrect (Gwet's AC1 statistic) or fail to correct for chance levels of agreement (the alpha, B, and delta approaches). Implications for diagnoses of psychiatric disorders are critically discussed. The importance of kappa is demonstrated in a number of clinical scenarios.