Measures of clinical agreement for nominal and categorical data: the kappa coefficient

Abstract

The desire to determine the extent inter-rater measurements obtained in a clinical setting are free from measurement error and reflect true scores has spurned a renewed interest in assessment of reliability. The kappa coefficient is considered the statistic of choice to analyze the reliability of nominal and categorical types of data recorded on the same patient by more than one clinician. This paper presents a simple computer program written in PASCAL that can be used in a clinical environment to quickly determine the reliability of nominal or categorical data. This computer program calculates both weighted and non-weighted kappa coefficients with their corresponding standard errors as well as bias-correcting jackknife estimates of kappa for use with small sample sizes.