Assessing inter-rater reliability for rating scales: resolving some basic issues

Abstract

This paper extends the recent work of Hall (1974) by presenting the minimal sample sizes and the specific linear agreement weights required for assessing the reliability of rating scales commonly used in neuropsychiatric and other clinico-medical settings. The weights are shown to vary as a function of (a) whether or not the rating scale contains a point of 'absence', and (b) the number of ordinal points on the scale.