Metrics of success: Measuring impact of a departmental near-miss incident learning system

Abstract

Purpose: There is a growing interest in the application of incident learning systems (ILS) to radiation oncology. The purpose of the present study is to define statistical metrics that may serve as benchmarks for successful operation of an incident learning system.

Methods and materials: A departmental safety and quality ILS was developed to monitor errors, near-miss events, and process improvement suggestions. Event reports were reviewed by a multiprofessional quality improvement committee. Events were scored by a near-miss risk index (NMRI) and categorized by event point of origination and discovery. Reporting trends were analyzed over a 2-year period, including total number and rates of events reported, users reporting, NMRI, and event origination and discovery.

Results: A total of 1897 reports were evaluated (1.0 reports/patient, 0.9 reports/unique treatment course). Participation in the ILS increased as demonstrated by total events (2.1 additional reports/month) and unique users (0.5 new users/month). Sixteen percent of reports had an NMRI of 0 (none), 42% had an NMRI of 1 (mild), 25% had an NMRI of 2 (moderate), 12% had an NMRI of 3 (severe), and 5% had an NMRI of 4 (critical). Event NMRI showed a significant decrease in the first 6 months (1.68-1.42, P < .001). Trends in origination and discovery of reports were broadly distributed between radiation therapy process steps and staff groups. The highest risk events originated in imaging for treatment planning (NMRI = 2.0 ± 1.1; P < .0001) and were detected in on-treatment quality management (NMRI = 1.7 ± 1.1; P = .003).

Conclusions: Over the initial 2-year period of ILS operation, rates of reporting increased, staff participation increased, and NMRI of reported events declined. These data mirror previously reported findings of improvement in safety culture endpoints. These metrics may be useful for other institutions seeking to create or evaluate their own ILS.