Behavior and interpretation of the kappa statistic: resolution of the two paradoxes

Abstract

Two apparent paradoxes have been identified for the kappa (kappa) statistic: (1) high levels of observer agreement with low kappa values; (2) lack of predictability of changes in kappa with changing marginals. The first paradox is a function of prevalence of the trait in the sample, while the second is related to symmetry of observations in the disagreement categories. While examining the behavior of kappa as a function of the distribution of responses in a contingency table, it was discovered that for any measured level of observer agreement (Po) there are three characteristic values of kappa: kappa max, kappa min, and kappa nor, each of which is a function only of Po. The characteristic values allow an observed kappa (kappa o) to be placed into perspective. By observing symmetry in agreement and disagreement categories, the behavior of kappa is readily understood and predictable. We define symmetry expressions for agreement (SA) and disagreement (SD) in order to represent and quantify these effects. Kappa alone has little interpretive value and we recommend that studies reporting kappa also report Po, SD, and P++ (agreement on the presence of the trait).