Measuring implementation strength: lessons from the evaluation of public health strategies in low- and middle-income settings

Abstract

Evaluation of strategies to ensure evidence-based, low-cost interventions reach those in need is critical. One approach is to measure the strength, or intensity, with which packages of interventions are delivered, in order to explore the association between implementation strength and public health gains. A recent systematic review suggested methodological guidance was needed. We described the approaches used in three examples of measures of implementation strength in evaluation. These addressed important public health topics with a substantial disease burden in low-and middle-income countries; they involved large-scale implementation; and featured evaluation designs without comparison areas. Strengths and weaknesses of the approaches were discussed. In the evaluation of Ethiopia's Health Extension Programme, implementation strength scoring for each kebele (ward) was based on aggregated data from interviews with mothers of children aged 12-23 months, reflecting their reports of contact with four elements of the programme. An evaluation of the Avahan HIV prevention programme in India used the cumulative amount of Avahan funding per HIV-infected person spent each year in each district. In these cases, a single measure was developed and the association with hypothesised programme outcomes presented. In the evaluation of the Affordable Medicines Facility-malaria, several implementation strength measures were developed based on the duration of activity of the programme and the level of implementation of supporting interventions. Measuring the strength of programme implementation and assessing its association with outcomes is a promising approach to strengthen pragmatic impact evaluation. Five key aspects of developing an implementation strength measure are to: (a) develop a logic model; (b) identify aspects of implementation to be assessed; (c) design and implement data collection from a range of data sources; (d) decide whether and how to combine data into a single measure; and, (e) plan whether and how to use the measure(s) in outcome analysis.

Keywords: Africa; Asia; Evaluation; HIV/AIDS; implementation strength; malaria; maternal; newborn and child health.

Figure 1.

Simplified logic model for the effect of Ethiopia’s health extension programme services on early initiation of breastfeeding.

Figure 2.

Simplified logic model for the effect of the Avahan initiative on HIV prevention in India at the district level.

Figure 3.

Simplified logic model for the effect of the AMFm on improved malaria treatment.