Infectious Disease Surveillance in the Big Data Era: Towards Faster and Locally Relevant Systems

Abstract

While big data have proven immensely useful in fields such as marketing and earth sciences, public health is still relying on more traditional surveillance systems and awaiting the fruits of a big data revolution. A new generation of big data surveillance systems is needed to achieve rapid, flexible, and local tracking of infectious diseases, especially for emerging pathogens. In this opinion piece, we reflect on the long and distinguished history of disease surveillance and discuss recent developments related to use of big data. We start with a brief review of traditional systems relying on clinical and laboratory reports. We then examine how large-volume medical claims data can, with great spatiotemporal resolution, help elucidate local disease patterns. Finally, we review efforts to develop surveillance systems based on digital and social data streams, including the recent rise and fall of Google Flu Trends. We conclude by advocating for increased use of hybrid systems combining information from traditional surveillance and big data sources, which seems the most promising option moving forward. Throughout the article, we use influenza as an exemplar of an emerging and reemerging infection which has traditionally been considered a model system for surveillance and modeling.

Keywords: Internet search queries; big data; death certificates; electronic patient records; infectious diseases surveillance; influenza; medical claims; real-time monitoring; syndromic data.

Figure 1.

Overview of the characteristics of infectious disease surveillance systems. Hybrid systems combining traditional surveillance with big data streams fall in the desirable zone associated with high information return and high data volume.