A methodological framework for the evaluation of syndromic surveillance systems: a case study of England

Abstract

Background: Syndromic surveillance complements traditional public health surveillance by collecting and analysing health indicators in near real time. The rationale of syndromic surveillance is that it may detect health threats faster than traditional surveillance systems permitting more timely, and hence potentially more effective public health action. The effectiveness of syndromic surveillance largely relies on the methods used to detect aberrations. Very few studies have evaluated the performance of syndromic surveillance systems and consequently little is known about the types of events that such systems can and cannot detect.

Methods: We introduce a framework for the evaluation of syndromic surveillance systems that can be used in any setting based upon the use of simulated scenarios. For a range of scenarios this allows the time and probability of detection to be determined and uncertainty is fully incorporated. In addition, we demonstrate how such a framework can model the benefits of increases in the number of centres reporting syndromic data and also determine the minimum size of outbreaks that can or cannot be detected. Here, we demonstrate its utility using simulations of national influenza outbreaks and localised outbreaks of cryptosporidiosis.

Results: Influenza outbreaks are consistently detected with larger outbreaks being detected in a more timely manner. Small cryptosporidiosis outbreaks (<1000 symptomatic individuals) are unlikely to be detected. We also demonstrate the advantages of having multiple syndromic data streams (e.g. emergency attendance data, telephone helpline data, general practice consultation data) as different streams are able to detect different outbreak types with different efficacy (e.g. emergency attendance data are useful for the detection of pandemic influenza but not for outbreaks of cryptosporidiosis). We also highlight that for any one disease, the utility of data streams may vary geographically, and that the detection ability of syndromic surveillance varies seasonally (e.g. an influenza outbreak starting in July is detected sooner than one starting later in the year). We argue that our framework constitutes a useful tool for public health emergency preparedness in multiple settings.

Conclusions: The proposed framework allows the exhaustive evaluation of any syndromic surveillance system and constitutes a useful tool for emergency preparedness and response.

Keywords: Cryptosporidiosis; Influenza; Scenarios; Simulation; Syndromic surveillance.

Fig. 1

Framework overview. Stages of the proposed framework for the evaluation of aberration detection methods

Fig. 2

Median time to detection per month. Median T_D for influenza and Cryptosporidium spp. outbreaks stratified by month of the onset and syndromic indicator. The colour of the boxes indicate the median T_D. The darkness of the box filling indicates the probability of detection (P_D) as defined in the legend. Boxes with a P_D≤0.50 were excluded

Fig. 3

Median T_D per level of access to healthcare. Median T_D for influenza outbreaks across four syndromic indicators for three different levels of access to healthcare (see Table 2). The dots indicate the median T_D, and the error bars depict the 95% studentized bootstrap prediction intervals. Prediction intervals were estimated using 1000 bootstrap samples. Estimates with a probability of detection ≤75% were excluded