The DataHarmonizer: a tool for faster data harmonization, validation, aggregation and analysis of pathogen genomics contextual information

Abstract

Pathogen genomics is a critical tool for public health surveillance, infection control, outbreak investigations as well as research. In order to make use of pathogen genomics data, they must be interpreted using contextual data (metadata). Contextual data include sample metadata, laboratory methods, patient demographics, clinical outcomes and epidemiological information. However, the variability in how contextual information is captured by different authorities and how it is encoded in different databases poses challenges for data interpretation, integration and their use/re-use. The DataHarmonizer is a template-driven spreadsheet application for harmonizing, validating and transforming genomics contextual data into submission-ready formats for public or private repositories. The tool's web browser-based JavaScript environment enables validation and its offline functionality and local installation increases data security. The DataHarmonizer was developed to address the data sharing needs that arose during the COVID-19 pandemic, and was used by members of the Canadian COVID Genomics Network (CanCOGeN) to harmonize SARS-CoV-2 contextual data for national surveillance and for public repository submission. In order to support coordination of international surveillance efforts, we have partnered with the Public Health Alliance for Genomic Epidemiology to also provide a template conforming to its SARS-CoV-2 contextual data specification for use worldwide. Templates are also being developed for One Health and foodborne pathogens. Overall, the DataHarmonizer tool improves the effectiveness and fidelity of contextual data capture as well as its subsequent usability. Harmonization of contextual information across authorities, platforms and systems globally improves interoperability and reusability of data for concerted public health and research initiatives to fight the current pandemic and future public health emergencies. While initially developed for the COVID-19 pandemic, its expansion to other data management applications and pathogens is already underway.

Keywords: contextual data; data management; genomic surveillance; harmonization; metadata.

Fig. 1.

The DataHarmonizer interface. (a) Users can access instructional videos, the reference guide and protocols for curation. Data providers can also access field-level guidance by double clicking on field headers. (b) Fields are colour-coded to indicate those that are required for Canadian SARS-CoV-2 surveillance (yellow), recommended (purple) and optional (grey). Users access different features, such as toggling between fields and automated filling of columns, via the control panel. (c) Validation of data highlights missing required information as well as errors which are highlighted in red. The ‘Next Error’ button enables users to scroll through and resolve errors systematically. (d) Users can export their data in different submission-ready formats.

Fig. 2.

Excerpt of a JSON file used to dynamically generate The DataHarmonizer application interface and functionality.

Fig. 3.

Validation of data. Missing or incorrect values are highlighted to better direct curation efforts. Curators can systematically address errors using the ‘Next Error’ button, which disappears when all issues have been addressed.

Fig. 4.

Customized exports automate data transformation for submission to a variety of third-party databases. (a) For Canadian national surveillance, contextual data can be exported to Canadian-specific portals and databases (e.g. VirusSeq Portal, CNPHI LaSER and NML LIMS) as well as international repositories (e.g. GISAID, NCBI). (b) Export from the PHA4GE template enables formatting for international databases.