Measurement and reliability: statistical thinking considerations

Abstract

Reliability is defined as the degree to which multiple assessments of a subject agree (reproducibility). There is increasing awareness among researchers that the two most appropriate measures of reliability are the intraclass correlation coefficient and kappa. However, unacceptable statistical measures of reliability such as chi-square, percent agreement, product moment correlation, as well as any measure of association and Yule's Y still appear in the literature. There are costs associated with improper measurements, unreliable diagnostic systems, inappropriate statistics and measures of reliability, and poor quality research. Costs are incurred when misleading information directs resources and talents into nonproductive avenues of research. The consequences of unreliable measurements and diagnosis are illustrated with some studies of schizophrenia.