Successes and Misses of Global Health Development: Detecting Temporal Concept Drift of Under-5 Mortality Prediction Models with Bias Scan

Abstract

Under-5 Mortality rates have been decreasing across Africa for the past two decades. Contributing factors include policy changes, technology, and health investments. This study identifies sub-populations that have experienced more-than-expected change in mortality rates (either increasing or decreasing) during this time period. We train under-5 mortality predictive models on Demographic and Health Survey (DHS) datasets from the early 2000s and apply those models to data collected in more recent versions of the survey. This provides an estimate of the risk current families would have faced in the past. We then apply techniques from anomalous pattern detection to identify sub-populations that have the most divergence between their predicted and observed mortality rates; higher and lower. These detected groups are examples of successes and possible misses of the health progress observed in Africa over the course of decades. Identifying these groups through data-driven discovery may lead to a better understanding of health policies in developing countries.

Figure 1:

Workflow Diagram for detecting Concept Drift between timesteps T0 and T1. Step 3 uses q’ which is the odds ratio of observed mortality between T1 and T0.

Figure 2:

Calibration Plot showing the data distribution shift in Ethiopia between T0 and T1. The T1 predictions are pessimistic, assigning higher probability of mortality to women/households that did not experience Under-5 mortality. Bias Scan is used to identify the sub-population where this divergence shows the most evidence of concept drift.