Pitfalls in the pursuit of objectivity: issues of reliability

Abstract

Objectivity has been one of the hallmarks in the assessment of clinical competence in recent decades. A consistent shift can be noticed in which subjective measures are being replaced by objective measurement methods. In the transition from subjective to objective methods trade-offs are involved, both in the effort expended and in the range of behaviours assessed. The issue of the presumed superiority of objective measures is addressed in two successive papers. In this paper a distinction is made between objectivity as a goal of measurement, marked by freedom of subjective influences in general, and objectivity as a set of strategies designed to reduce measurement error. The latter has been termed objectification. The central claim of this paper is that these two approaches to assessment do not necessarily coincide. By reviewing a number of studies comparing subjective and objectified measurement methods, the claim of the supremacy of the latter with respect to reliability is discussed. The results of these studies indicate that objectified methods do not inherently provide more reliable scores. Objectified methods may even provide unwanted outcomes, such as negative effects on study behaviour and triviality of the content being measured. The latter issues, related to validity, efficiency and acceptability, are discussed in a second paper.