The Johns Hopkins University Center for Systems Science and Engineering COVID-19 Dashboard: data collection process, challenges faced, and lessons learned

Abstract

On Jan 22, 2020, a day after the USA reported its first COVID-19 case, the Johns Hopkins University Center for Systems Science and Engineering (JHU CSSE) launched the first global real-time coronavirus surveillance system: the JHU CSSE COVID-19 Dashboard. As of June 1, 2022, the dashboard has served the global audience for more than 30 consecutive months, totalling over 226 billion feature layer requests and 3·6 billion page views. The highest daily record was set on March 29, 2020, with more than 4·6 billion requests and over 69 million views. This Personal View reveals the fundamental technical details of the entire data system underlying the dashboard, including data collection, data fusion logic, data curation and sharing, anomaly detection, data corrections, and the human resources required to support such an effort. The Personal View also covers the challenges, ranging from data visualisation to reporting standardisation. The details presented here help develop a framework for future, large-scale public health-related data collection and reporting.

Figure 1

Johns Hopkins University Center for Systems Science and Engineering Dashboard usage and milestones, including the number of requests and views Requests refer to the number of times a visitor interacts with the dashboard system, such as clicking on a specific country. Views refer to the number of times the dashboard, either desktop or mobile versions, is loaded on the visitor's end. Global COVID-19 cases and deaths in dashed lines are reference plots. (A, B) Total usage from Jan 21, 2020, to June 1, 2022. (C, D) Daily usage before June 15, 2020. Daily cases and daily deaths are smoothed by 7-day moving average.

Figure 2

Graphical summary of the Johns Hopkins University Center for Systems Science and Engineering Dashboard data pipeline The pipeline can be separated into four main steps. (A) Data sourcing describes the identification and validation of trusted, open-source data sources. (B) Autonomous collection uses web scraping algorithms to collect raw data from open-source data sources. (C) Comprehensive data curation passes the data through several quality control mechanisms including an in-house designed anomaly detection service. Data fusion services curate the cleaned data into a single production database. (D) Data sharing is the publication of production data into our online data products.

Figure 3

Evolution of the dashboard visualisation (A) Initially, our efforts were focused on the spread of cases in China. (B) As the virus spread globally, the default view was expanded to include the entire world. (C) In the most current version, vaccination data has been added and the time series has been adjusted from daily to weekly bars.