Reliability study of the European appropriateness evaluation protocol

Abstract

Objective: To help to co-ordinate and harmonize research on utilization review in Europe, the US Appropriateness Evaluation Protocol (f¿EP) was adapted for use in the European setting. The aim of this paper is to assess the reliability of the European version of the AEP (EU-AEP).

Design: Nineteen English-language medical records were reviewed by a physician reviewer from each of six participating countries: Austria, France, Italy, Spain, Switzerland and the UK. Each of the six reviewers was asked to assess the appropriateness of the 19 admissions and 31 hospitalization days (19 admission days and 12 randomly selected days of hospital stay, excluding days of discharge) using the revised review instrument. To evaluate inter-rater reliability, the kappa statistic was used to measure overall and pair-wise agreement for the assessment of appropriateness of admission and of day of care, respectively.

Results: For admission, the overall kappa statistic among the six reviewers was 0.64, with kappa values for each pair of reviewers in the range 0.46-0.86. For day of care, the kappa was 0.59, with pair-wise kappa coefficients in the range 0.25-0.95.

Conclusion: The observed agreement could be considered substantial, especially if the fact that medical records were hand-written in a language native to only one of the reviewers is considered. Besides all the study limitations, this finding provides at least preliminary support for the application of the EU-AEP as a reliable instrument in the European setting, including application in comparative studies involving two or more countries.