Hierarchical Statistical Models to Represent and Visualize Survey Evidence for Program Evaluation: iCCM in Malawi

Abstract

Policy and Program evaluation for maternal, newborn and child health is becoming increasingly complex due to changing contexts. Monitoring and evaluation efforts in this area can take advantage of large nationally representative household surveys such as DHS or MICS that are increasing in size and frequency, however, this analysis presents challenges on several fronts. We propose an approach with hierarchical models for cross-sectional survey data to describe evidence relating to program evaluation, and apply this approach to the recent scale up of iCCM in Malawi. We describe careseeking for children sick with diarrhea, pneumonia, or malaria with empirical Bayes estimates for each district of Malawi at two time points, both for careseeking from any source, and for careseeking only from health surveillance assistants (HSA). We do not find evidence that children in areas with more HSA trained in iCCM are more likely to seek care for pneumonia, diarrhea, or malaria, despite evidence that many indeed are seeking care from HSA. Children in areas with more HSA trained in iCCM are more likely to seek care from a HSA, with 100 additional trained health workers in a district corresponding to a 2% average increase in careseeking from HSA. The hierarchical models presented here provide a flexible set of methods that describe the primary evidence for evaluating iCCM in Malawi and which could be extended to formal causal analyses, and to analysis for other similar evaluations with national survey data.

Fig 1. Evaluation framework.

Conceptual framework for the evaluation of CCM in Malawi, with careseeking in 2014 as the primary outcome of interest, represented as actual (C₁*) and measured (C₁). Baseline (2010) careseeking is represented as actual (C₀*) and measured (C₀). We assume that careseeking in 2014 is related to careseeking in 2010 and program implementation of iCCM (P). Program implementation may be determined by baseline careseeking, and potentially by unobserved factor U*.

Fig 2. District careseeking from HSA.

Estimated careseeking from HSA for children 2–59 months old among those sick with pneumonia, diarrhea, or malaria in the 2010 Demographic and Health Survey (a) and the 2014 Multiple Indicator Cluster Survey (b) and the change from 2010 to 2014 (c) by district, as well as the number of Health Surveillance Assistants active in each district (d).

Fig 3. District careseeking from all sources.

Estimated careseeking from all sources for children 2–59 months old among those sick with pneumonia, diarrhea, or malaria in the 2010 Demographic and Health Survey (a) and the 2014 Multiple Indicator Cluster Survey (b) by district, and the change from 2010 to 2014 (c), with the number of Health Surveillance Assistants active in each district (d).