Interpreting kappa in observational research: baserate matters

Abstract

Kappa (Cohen, 1960) is a popular agreement statistic used to estimate the accuracy of observers. The response of kappa to differing baserates was examined and methods for estimating the accuracy of observers presented. Results suggest that setting a single value of kappa as "minimally acceptable" (i.e., a criterion value) is not useful in ensuring adequate accuracy of observers. Instead, researchers should use the best estimate of the true baserate of the target behavior and the obtained kappa to estimate the accuracy of observers. Investigators can then compare the estimated accuracy of observers to a preselected criterion level. Guidelines are given for selecting a criterion accuracy level.