Exploring Changes to the Actionability of COVID-19 Dashboards Over the Course of 2020 in the Canadian Context: Descriptive Assessment and Expert Appraisal Study

Abstract

Background: Public web-based COVID-19 dashboards are in use worldwide to communicate pandemic-related information. Actionability of dashboards, as a predictor of their potential use for data-driven decision-making, was assessed in a global study during the early stages of the pandemic. It revealed a widespread lack of features needed to support actionability. In view of the inherently dynamic nature of dashboards and their unprecedented speed of creation, the evolution of dashboards and changes to their actionability merit exploration.

Objective: We aimed to explore how COVID-19 dashboards evolved in the Canadian context during 2020 and whether the presence of actionability features changed over time.

Methods: We conducted a descriptive assessment of a pan-Canadian sample of COVID-19 dashboards (N=26), followed by an appraisal of changes to their actionability by a panel of expert scorers (N=8). Scorers assessed the dashboards at two points in time, July and November 2020, using an assessment tool informed by communication theory and health care performance intelligence. Applying the nominal group technique, scorers were grouped in panels of three, and evaluated the presence of the seven defined features of highly actionable dashboards at each time point.

Results: Improvements had been made to the dashboards over time. These predominantly involved data provision (specificity of geographic breakdowns, range of indicators reported, and explanations of data sources or calculations) and advancements enabled by the technologies employed (customization of time trends and interactive or visual chart elements). Further improvements in actionability were noted especially in features involving local-level data provision, time-trend reporting, and indicator management. No improvements were found in communicative elements (clarity of purpose and audience), while the use of storytelling techniques to narrate trends remained largely absent from the dashboards.

Conclusions: Improvements to COVID-19 dashboards in the Canadian context during 2020 were seen mostly in data availability and dashboard technology. Further improving the actionability of dashboards for public reporting will require attention to both technical and organizational aspects of dashboard development. Such efforts would include better skill-mixing across disciplines, continued investment in data standards, and clearer mandates for their developers to ensure accountability and the development of purpose-driven dashboards.

Keywords: COVID-19; Canada; assessment; communication; dashboards; decision-making; development; dynamic; health information management; medical informatics; performance measures; public health; public reporting of health care data; qualitative research; surveillance.

Figure 1

Distribution of COVID-19 dashboards sampled and types of organizations responsible for their development. Circles denote municipal-level dashboards included in the sample, and the colors denote the respective organization types. These dashboards are counted in the tally shown per jurisdiction. The Public Health Agency of Canada’s COVID-19 dashboard is hosted on the federal Government of Canada webpage. In other instances, dashboards developed by public health authorities are hosted on dedicated webpages.

Figure 2

Change in actionability across dashboards (n=26) over time in 2020. Not present: the feature is not found on the dashboard; somewhat present: some elements of the feature are present on the dashboard but room for improvement; present: the specific feature is clearly demonstrated and a good practice example of the feature is present. See Multimedia Appendix 1 for full scoring details and Multimedia Appendix 3 for the level of agreement between panel members.