Strengths, pitfalls, and lessons learned in implementing electronic collection of childhood vaccination data in Zambia: The SmartCare experience

Abstract

Background: Despite widespread interest in computerized vaccination information systems, evaluation of the data quality in these systems and their acceptability to frontline healthcare workers in low and middle-income countries aren't well addressed in the literature.

Objectives: Evaluation of vaccination data quality and facility-level staff perspectives on the strengths and challenges of a vaccination data module in a widely used electronic health record (EHR) system in Zambia.

Methods: After a desk review of data from two provinces, a cross-sectional mixed methods study was designed, including quantitative analysis of data quality and qualitative analysis of the module's acceptability to facility staff, using the Information System Success model as the framework for evaluation of system quality, service quality, and information quality. Data were collected from 10 purposively sampled health facilities.

Results: There was low current use of the vaccination module by facilities in the study area (2%). Daily power outages presented a practical challenge. Staff who had used previous EHRs had concerns about sustainability.

System quality: While the module was user-friendly, there were concerns about EHR compatibility with vaccination workflow and outreach settings, where vaccines are commonly administered to older children.

Service quality: The module was viewed as dependable; perceptions were influenced by computer literacy.

Information quality: The database contained incomplete and incongruous data. Staff perceived data as accurate but incomplete; easy access to data was a strength.

Conclusions: Potential benefits of the vaccination module were frequently unrealized due to infrastructure, workflow, and data flow challenges that resulted in low module use and poor information quality. Elements to optimize vaccination information system implementation could include robust engagement of facility-level staff in system design, system suitability to the vaccination setting and workflow, and comprehensive planning for data flow, sustainability, data monitoring and feedback. Adaptability to the outreach setting might be increasingly important as vaccination schedules extend past infancy.

Keywords: Data collection; Data quality; Electronic health records; Registries; Vaccination records.

Fig. 1.

Comparison between the number of vaccinations (Penta1, Penta3, and Measles1) recorded on facility monthly report and facility SmartCare for selected facilities in Lusaka and Southern Province—Zambia, Oct-Dec 2015 Dotted diagonal line represents expected location of data points if data in the traditional paper facility monthly report and facility SmartCare database were a perfect match. Each data point represents the doses recorded for a single antigen over a single month at a health facility simultaneously using traditional paper records and SmartCare records for vaccination data. Data points above the line indicate that more doses were recorded on the facility monthly report than in the facility SmartCare database.

Fig. 2.

Comparison between the number of vaccinations (Penta1, Penta3, and Measles1) recorded in the national SmartCare and facility SmartCare for selected facilities in Lusaka and Southern Province—Zambia, Oct-Dec 2015 Each data point represents the doses recorded for a single antigen studied in a single month at one facility. Data points above the dotted line indicate that there were a greater number of doses in the national database as compared to the facility database. Data points below indicate that there are more doses recorded in the facility database than those reflected in the national database.