Finding leading indicators for disease outbreaks: filtering, cross-correlation, and caveats

Abstract

Bioterrorism and emerging infectious diseases such as influenza have spurred research into rapid outbreak detection. One primary thrust of this research has been to identify data sources that provide early indication of a disease outbreak by being leading indicators relative to other established data sources. Researchers tend to rely on the sample cross-correlation function (CCF) to quantify the association between two data sources. There has been, however, little consideration by medical informatics researchers of the influence of methodological choices on the ability of the CCF to identify a lead-lag relationship between time series. We draw on experience from the econometric and environmental health communities, and we use simulation to demonstrate that the sample CCF is highly prone to bias. Specifically, long-scale phenomena tend to overwhelm the CCF, obscuring phenomena at shorter wave lengths. Researchers seeking lead-lag relationships in surveillance data must therefore stipulate the scale length of the features of interest (e.g., short-scale spikes versus long-scale seasonal fluctuations) and then filter the data appropriately--to diminish the influence of other features, which may mask the features of interest. Otherwise, conclusions drawn from the sample CCF of bi-variate time-series data will inevitably be ambiguous and often altogether misleading.

Figure 1

Typical aggregate diagnostic and OTC sales data: Summer minimum through Winter Maximum.

Figure 2

Composite data and raw cross-correlation; dominated by relations on long scale.

Figure 3

Cross-correlation of composite signals after first differencing; peak at −6 is now manifest, maximum at +4 due to long-term trends still present.

Figure 4

Lag Correlation after second differencing completely eliminates the contribution due to long-scale fluctuations, and fully reveals relationship at −6 days due to short scale components.

Figure 5

(A to F from top to bottom) (A) Two simulated aggregated clinical respiratory data series; multiplicative model with seasonal and 7-day cycle and independent log-normal random components; with a 5-day “outbreak” signature in both series, lagged by 10 days. (B) 7-day cyclic component eliminated by 7-day moving average. (C) 7-day cyclic component eliminated by 7-day lagged difference. (D) Cross-correlation function of the untreated pair of data series (A). No evidence of the outbreak at lag 10. (E) Cross-correlation function (negative-log complement scale) of the 7-day moving average data series (B). CCF dominated by the common long-scale seasonal component. No evidence of outbreak at lag 10. (F) CCF of the 7-day lag difference data series. Shows evidence of the alignment at lag 10 due to the short-scale outbreak signatures.