A new method for assessing interexaminer agreement when multiple ratings are made on a single subject: applications to the assessment of neuropsychiatric symtomatology

Abstract

A new method is introduced for assessing levels of interexaminer agreement when multiple ratings are made on a single subject, with an application in psychiatric research. It is designed to provide an overall level of interexaminer agreement and separate indices of agreement for each examiner. These indices are based on biostatistical and clinical criteria to determine whether the ratings of any given examiner are appreciably higher or lower than the group average, or a consensus diagnosis. A number of examples, from ongoing psychiatric research, are provided to illustrate conditions favoring the application of the new methodology. Finally, the necessary software for performing the analyses is available to clinical investigators with interest in this area of assessment.