Global Variations in Event-Based Surveillance for Disease Outbreak Detection: Time Series Analysis

Abstract

Background: Robust and flexible infectious disease surveillance is crucial for public health. Event-based surveillance (EBS) was developed to allow timely detection of infectious disease outbreaks by using mostly web-based data. Despite its widespread use, EBS has not been evaluated systematically on a global scale in terms of outbreak detection performance.

Objective: The aim of this study was to assess the variation in the timing and frequency of EBS reports compared to true outbreaks and to identify the determinants of variability by using the example of seasonal influenza epidemic in 24 countries.

Methods: We obtained influenza-related reports between January 2013 and December 2019 from 2 EBS systems, that is, HealthMap and the World Health Organization Epidemic Intelligence from Open Sources (EIOS), and weekly virological influenza counts for the same period from FluNet as the gold standard. Influenza epidemic periods were detected based on report frequency by using Bayesian change point analysis. Timely sensitivity, that is, outbreak detection within the first 2 weeks before or after an outbreak onset was calculated along with sensitivity, specificity, positive predictive value, and timeliness of detection. Linear regressions were performed to assess the influence of country-specific factors on EBS performance.

Results: Overall, while monitoring the frequency of EBS reports over 7 years in 24 countries, we detected 175 out of 238 outbreaks (73.5%) but only 22 out of 238 (9.2%) within 2 weeks before or after an outbreak onset; in the best case, while monitoring the frequency of health-related reports, we identified 2 out of 6 outbreaks (33%) within 2 weeks of onset. The positive predictive value varied between 9% and 100% for HealthMap and from 0 to 100% for EIOS, and timeliness of detection ranged from 13% to 94% for HealthMap and from 0% to 92% for EIOS, whereas system specificity was generally high (59%-100%). The number of EBS reports available within a country, the human development index, and the country's geographical location partially explained the high variability in system performance across countries.

Conclusions: We documented the global variation of EBS performance and demonstrated that monitoring the report frequency alone in EBS may be insufficient for the timely detection of outbreaks. In particular, in low- and middle-income countries, low data quality and report frequency impair the sensitivity and timeliness of disease surveillance through EBS. Therefore, advances in the development and evaluation and EBS are needed, particularly in low-resource settings.

Keywords: analysis; data; detect; detection; digital disease detection; disease; epidemic; event-based surveillance; infectious disease outbreak; influenza; outbreak; public health; public health surveillance; surveillance.

Figure 1

Illustration of the workflow. BCP: Bayesian change point analysis; EBS: event-based surveillance.

Figure 2

Time series of weekly reports relating to influenza from Epidemic Intelligence from Open Sources and HealthMap and weekly virological influenza counts from FluNet for selected countries from January 2013 to December 2019. Epidemic periods found with Bayesian change point analysis for each system are highlighted in red, and nonepidemic periods are shown in grey. EIOS: Epidemic Intelligence from Open Sources.

Figure 3

HealthMap and Epidemic Intelligence from Open Sources performance metrics for the detection of influenza outbreaks from January 2013 to July 2019 (HealthMap) or from November 2017 to December 2019 (Epidemic Intelligence from Open Sources). All metrics were calculated with FluNet data as reference. EIOS: Epidemic Intelligence from Open Sources.