Inter-rater variability in scoring of Addenbrooke's Cognitive Examination-Third Edition (ACE-III) protocols

Abstract

Background: Despite its wide use in dementia diagnosis on the basis of cut-off points, the inter-rater variability of the Addenbrooke's Cognitive Examination-Third Edition (ACE-III) has been poorly studied.

Methods: Thirty-one healthcare professionals from an older adults' mental health team scored two ACE-III protocols based on mock patients in a computerised form. Scoring accuracy, as well as total and domain-specific scoring variability, were calculated; factors relevant to participants were obtained, including their level of experience and self-rated confidence administering the ACE-III.

Results: There was considerable inter-rater variability (up to 18 points for one of the cases), and one case's mean score was significantly higher (by nearly four points) than the true score. The Fluency, Visuospatial and Attention domains had greater levels of variability than Language and Memory. Higher scoring accuracy was not associated with either greater levels of experience or higher self-confidence in administering the ACE-III.

Conclusions: The results suggest that the ACE-III is susceptible to scoring error and considerable inter-rater variability, which highlights the critical importance of initial, and continued, administration and scoring training.

Keywords: ACE-III; dementia; inter-rater variability; scoring.