[VII: Diagnostic trials: Simple measures of validity and reliability]

Abstract

The evaluation of a new diagnostic or imaging device has to focus both on its diagnostic validity and reliability. Cohen's kappa coefficient can be used to assess the agreement of the new device's findings with those of an established diagnostic reference, it therefore presents a surrogate measure of validity. Additional measures of validity are sensitivity and specificity, which assess the order of agreement with the reference for positive and negative reference findings, respectively. The latter, however, can only be applied for binary reference findings. If discordant findings are observed, the McNemar test can be used to detect a diagnostic shift between diagnostic novum and reference. The estimation of reliability can be based on the independent parallel rating of a patient subset by two clinical readers; again the kappa coefficient measures the agreement of the parallel ratings and therefore represents a surrogate measure of reliability. The McNemar test can be used to detect an observer bias.