A comparison model for measuring individual agreement

Abstract

This article proposes a general comparison model for assessing individual agreement of k ≥ 2 raters evaluating n subjects with m ≥ 2 replicated readings. Users can explore total-rater agreement relative to intrarater agreement where any subset of the k raters can be selected in the numerator and denominator. Users are also allowed to compare intrarater agreement among selected raters. Based on the ratio of mean squared deviations (MSDs), two comparative agreement indices, total-intra ratio (TIR) and intra-intra ratio (IIR), are proposed. The TIR is a noninferiority assessment such that the differences of individual readings from different raters cannot be inferior by a prespecified margin to the differences of the replicated readings within raters. TIR can be used whether a reference exists or not. The method used by the Food and Drug Administration (FDA) for evaluating individual bioequivalence under relative scale becomes the special case of our approach. The IIR is a classical assessment such that the precision of selected raters can be better than; equal to; or worse than that of other raters. The estimation and statistical inference of TIR and IIR are obtained through GEE methodology.