Surgical data strengthening in Ethiopia: results of a Kirkpatrick framework evaluation of a data quality intervention

Abstract

Background: One key challenge in improving surgical care in resource-limited settings is the lack of high-quality and informative data. In Ethiopia, the Safe Surgery 2020 (SS2020) project developed surgical key performance indicators (KPIs) to evaluate surgical care within the country. New data collection methods were developed and piloted in 10 SS2020 intervention hospitals in the Amhara and Tigray regions of Ethiopia. Objective: To assess the feasibility of collecting and reporting new surgical indicators and measure the impact of a surgical Data Quality Intervention (DQI) in rural Ethiopian hospitals. Methods: An 8-week DQI was implemented to roll-out new data collection tools in SS2020 hospitals. The Kirkpatrick Method, a widely used mixed-method evaluation framework for training programs, was used to assess the impact of the DQI. Feedback surveys and focus groups at various timepoints evaluated the impact of the intervention on surgical data quality, the feasibility of a new data collection system, and the potential for national scale-up. Results: Results of the evaluation are largely positive and promising. DQI participants reported knowledge gain, behavior change, and improved surgical data quality, as well as greater teamwork, communication, leadership, and accountability among surgical staff. Barriers remained in collection of high-quality data, such as lack of adequate human resources and electronic data reporting infrastructure. Conclusions: Study results are largely positive and make evident that surgical data capture is feasible in low-resource settings and warrants more investment in global surgery efforts. This type of training and mentorship model can be successful in changing individual behavior and institutional culture regarding surgical data collection and reporting. Use of the Kirkpatrick Framework for evaluation of a surgical DQI is an innovative contribution to literature and can be easily adapted and expanded for use within global surgery.

Keywords: Ethiopia; Global surgery; Kirkpatrick evaluation; monitoring and evaluation in safe surgery; safe surgery.

Figure 1.

Amhara and Tigray pre- and post-test scores comparison (Kirkpatrick level 2)

Figure 2.

Percentage of participants who agree and strongly agree with program success, individual behavior change, and institutional behavior change items following the intervention (Kirkpatrick levels 3 & 4)