Operationalizing and automating Data Governance

Abstract

The ability to cross data from multiple sources represents a competitive advantage for organizations. Yet, the governance of the data lifecycle, from the data sources into valuable insights, is largely performed in an ad-hoc or manual manner. This is specifically concerning in scenarios where tens or hundreds of continuously evolving data sources produce semi-structured data. To overcome this challenge, we develop a framework for operationalizing and automating data governance. For the first, we propose a zoned data lake architecture and a set of data governance processes that allow the systematic ingestion, transformation and integration of data from heterogeneous sources, in order to make them readily available for business users. For the second, we propose a set of metadata artifacts that allow the automatic execution of data governance processes, addressing a wide range of data management challenges. We showcase the usefulness of the proposed approach using a real world use case, stemming from the collaborative project with the World Health Organization for the management and analysis of data about Neglected Tropical Diseases. Overall, this work contributes on facilitating organizations the adoption of data-driven strategies into a cohesive framework operationalizing and automating data governance.

Keywords: Big Data; Data Governance; Data Integration; Metadata.

Fig. 1

The organization data assets from [4] (in gray those covered by this paper)

Fig. 2

WISCENTD Use Case

Fig. 3

High-level overview of the proposed Data Lake architecture and its zones

Fig. 4

Example of the Temporal Landing for the NTD-related data

Fig. 5

Persistent landing for medicine request data

Fig. 6

Example of the schema for the tables in Formatted Zone for the WIMEDS data

Fig. 7

Example of the medicine request dashboards in the Exploitation Zone

Fig. 8

UML class diagram for the metadata artifacts

Fig. 9

Metadata artifacts and their attributes

Fig. 10

Instantiation of the system architecture for WISCENTD use case

Fig. 11

Data Collectors DGP

Fig. 12

Execution of the Data Collectors DGP (WISCENTD Use Case)

Fig. 13

Data Persistence Loaders DGP

Fig. 14

Execution of the Data Persistence Loader DGP (WISCENTD Use Case)

Fig. 15

Data Formatters DGP

Fig. 16

Execution of the Data Formatters DGP (WISCENTD Use Case)