A review of the reliability and validity of OPCRIT in relation to its use for the routine clinical assessment of mental health patients

Abstract

The OPCRIT program is a symptom checklist with accompanying algorithms producing operationally defined diagnoses. We undertook a review of studies which had used OPCRIT and had reported statistics concerning its reliability and validity, producing summary measures from 44 studies. The first main measure of interest was inter-rater reliability where mean kappa values indicated that agreement between raters was "substantial" with a marginal improvement at the diagnostic (0.76) versus individual item (0.69) level. The second main measure of interest was convergent validity - the agreement between OPCRIT and clinical diagnoses. Most studies reported these figures as concordance rates suggesting mean agreement, unadjusted for chance, of 69%. Very few studies used the chance-adjusted kappa statistic but where this was used agreement was "fair" (0.39). Agreement between OPCRIT and other research diagnoses was "moderate" (0.60). We also considered differences between the way OPCRIT has traditionally been used in research settings and the naturalistic manner in which it will be employed in the hospital ward. This review provides a summary of the reliability and validity of OPCRIT, which will be considered during the preparation for its use in the routine characterization of mental health patients in clinical settings.

Figure 1

The reliability and validity of OPCRIT as represented by average kappa bandings from 32 studies. X axis categorized into: Category 1, Agreement between raters; Category 2, Agreement between OPCRIT and clinical diagnoses; Category 3, Agreement between OPCRIT diagnoses and other diagnostic (research) methods; Category 4, Agreement between OPCRIT diagnoses on the same patients but based on different information sources; Category 5, Agreement between repeated OPCRIT ratings on the same patients. Y axis categorized into Landis and Koch's (1977) proposed “agreement” bandings. Where applicable, ratings are split into diagnostic (solid line) and item level (dashed line) agreement. All values are means except item‐level agreement for Category 5 which is an approximation. Values represent a mix of classificatory systems (i.e. DSM, ICD and RDC).