Inter- and intra-rater reliability for classification of medication related events in paediatric inpatients

Abstract

Background: In medication safety research studies medication related events are often classified by type, seriousness, and degree of preventability, but there is currently no universally reliable "gold standard" approach. The reliability (reproducibility) of this process is important as the targeting of prevention strategies is often based on specific categories of event. The aim of this study was to determine the reliability of reviewer judgements regarding classification of paediatric inpatient medication related events.

Methods: Three health professionals independently reviewed suspected medication related events and classified them by type (adverse drug event (ADE), potential ADE, medication error, rule violation, or other event). ADEs and potential ADEs were then rated according to seriousness of patient injury using a seven point scale and preventability using a decision algorithm and a six point scale. Inter- and intra-rater reliabilities were calculated using the kappa (kappa) statistic.

Results: Agreement between all three reviewers regarding event type ranged from "slight" for potential ADEs (kappa = 0.20, 95% CI 0.00 to 0.40) to "substantial" agreement for the presence of an ADE (kappa = 0.73, 95% CI 0.69 to 0.77). Agreement ranged from "slight" (kappa = 0.06, 95% CI 0.02 to 0.10) to "fair" (kappa = 0.34, 95% CI 0.30 to 0.38) for seriousness classifications but, by collapsing the seven categories into serious versus not serious, "moderate" agreement was found (kappa = 0.50, 95% CI 0.46 to 0.54). For preventability decision, overall agreement was "fair" (kappa = 0.37, 95% CI 0.33 to 0.41) but "moderate" for not preventable events (kappa = 0.47, 95% CI 0.43 to 0.51).

Conclusion: Trained reviewers can reliably assess paediatric inpatient medication related events for the presence of an ADE and for its seriousness. Assessments of preventability appeared to be a more difficult judgement in children and approaches that improve reliability would be useful.