Reliability and validity of utilization review criteria. Appropriateness Evaluation Protocol, Standardized Medreview Instrument, and Intensity-Severity-Discharge criteria

Abstract

A study was conducted to assess the reliability and validity of the Appropriateness Evaluation Protocol (AEP), the Standardized Medreview Instrument (SMI) and the Intensity-Severity-Discharge criteria set (ISD), three utilization review instruments used to determine whether inpatient care is required. Reliability and validity were assessed for retrospective application of these instruments to charts of a sample of 119 medical cases from 21 hospitals in the state of Michigan. The reliability of each instrument was determined by having the instrument applied by two different nurse reviewers to each hospital record. Results indicated that the AEP and ISD were moderately reliable, while the SMI had low reliability. The validity of each instrument was tested by comparing the judgments of nurse reviewers using the instruments with the judgment of a panel of physicians. The AEP and ISD were found to be moderately valid and the SMI was found to have low validity. Results suggested that the SMI should not be used. The modest level of validity of the other two instruments suggests that payment should never be denied on the basis of the instrument alone. Payment should be denied only if a physician confirms the judgment based on the instrument that inpatient care was not required.