A Simulation-Based Study on the Comparison of Statistical and Time Series Forecasting Methods for Early Detection of Infectious Disease Outbreaks

Abstract

Early detection of infectious disease outbreaks is one of the important and significant issues in syndromic surveillance systems. It helps to provide a rapid epidemiological response and reduce morbidity and mortality. In order to upgrade the current system at the Korea Centers for Disease Control and Prevention (KCDC), a comparative study of state-of-the-art techniques is required. We compared four different temporal outbreak detection algorithms: the CUmulative SUM (CUSUM), the Early Aberration Reporting System (EARS), the autoregressive integrated moving average (ARIMA), and the Holt-Winters algorithm. The comparison was performed based on not only 42 different time series generated taking into account trends, seasonality, and randomly occurring outbreaks, but also real-world daily and weekly data related to diarrhea infection. The algorithms were evaluated using different metrics. These were namely, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), F1 score, symmetric mean absolute percent error (sMAPE), root-mean-square error (RMSE), and mean absolute deviation (MAD). Although the comparison results showed better performance for the EARS C3 method with respect to the other algorithms, despite the characteristics of the underlying time series data, Holt⁻Winters showed better performance when the baseline frequency and the dispersion parameter values were both less than 1.5 and 2, respectively.

Keywords: aberration detection; outbreak detection; syndromic diarrhea; syndromic surveillance.

Figure 1

Framework for comparing the outbreak detection algorithms. EARS: Early Aberration Reporting System; ARIMA: auto regressive integrated moving average; PPV: positive predictive value; NPV: negative predictive value; sMAPE: symmetric mean absolute percent error; RMSE: root-mean-square error; MAD: mean absolute deviation; CUSUM: CUmulative SUM.

Figure 2

Examples of simulated data: (a) Scenario 8; (b) Scenario 10; (c) Scenario 12; (d) Scenario 13; (e) Scenario 15; and (f) Scenario 17. In each graph, the x-axis represents time, the y-axis represents the number of infections, and the green-colored plus sign resembles outbreaks.

Figure 3

(a) The results of the four different CUSUM algorithms with different sets of parameters in terms of sensitivity, specificity, PPV, and NPV across the all scenarios. (b) The results of the four different CUSUM algorithms with different sets of parameters in terms of F1 score, sMAPE, RMSE, and MAD across the all scenarios. CUSUM: the CUmulative SUM.

Figure 3

(a) The results of the four different CUSUM algorithms with different sets of parameters in terms of sensitivity, specificity, PPV, and NPV across the all scenarios. (b) The results of the four different CUSUM algorithms with different sets of parameters in terms of F1 score, sMAPE, RMSE, and MAD across the all scenarios. CUSUM: the CUmulative SUM.

Figure 4

(a) The results of comparing EARS C1, C2, and C3 in terms of sensitivity, specificity, PPV, and NPV across the all scenarios. EARS C3 showed better performance in almost all cases. (b) The results of comparing EARS C1, C2, and C3 in terms of F1 score, sMAPE, RMSE, and MAD across the all scenarios. EARS C3 showed better performance in almost all cases as it showed lower values of sMAPE, RMSE, and MAD, and higher values of F1 score.

Figure 4

(a) The results of comparing EARS C1, C2, and C3 in terms of sensitivity, specificity, PPV, and NPV across the all scenarios. EARS C3 showed better performance in almost all cases. (b) The results of comparing EARS C1, C2, and C3 in terms of F1 score, sMAPE, RMSE, and MAD across the all scenarios. EARS C3 showed better performance in almost all cases as it showed lower values of sMAPE, RMSE, and MAD, and higher values of F1 score.

Figure 5

(a) The results of comparing CUSUM, EARS C3, ARIMA, and Holt-Winters in terms of sensitivity, specificity, PPV, and NPV across the all scenarios. (b) The results of comparing CUSUM, EARS C3, ARIMA, and Holt-Winters in terms of F1 score, sMAPE, RMSE, and MAD across the all scenarios.

Figure 5

(a) The results of comparing CUSUM, EARS C3, ARIMA, and Holt-Winters in terms of sensitivity, specificity, PPV, and NPV across the all scenarios. (b) The results of comparing CUSUM, EARS C3, ARIMA, and Holt-Winters in terms of F1 score, sMAPE, RMSE, and MAD across the all scenarios.