Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022;9(1):117.
doi: 10.1186/s40537-022-00673-5. Epub 2022 Dec 10.

Operationalizing and automating Data Governance

Sergi Nadal  1 , Petar Jovanovic  1 , Besim Bilalli  1 , Oscar Romero  1
Affiliations

Operationalizing and automating Data Governance

Sergi Nadal et al. J Big Data. 2022.

Abstract

The ability to cross data from multiple sources represents a competitive advantage for organizations. Yet, the governance of the data lifecycle, from the data sources into valuable insights, is largely performed in an ad-hoc or manual manner. This is specifically concerning in scenarios where tens or hundreds of continuously evolving data sources produce semi-structured data. To overcome this challenge, we develop a framework for operationalizing and automating data governance. For the first, we propose a zoned data lake architecture and a set of data governance processes that allow the systematic ingestion, transformation and integration of data from heterogeneous sources, in order to make them readily available for business users. For the second, we propose a set of metadata artifacts that allow the automatic execution of data governance processes, addressing a wide range of data management challenges. We showcase the usefulness of the proposed approach using a real world use case, stemming from the collaborative project with the World Health Organization for the management and analysis of data about Neglected Tropical Diseases. Overall, this work contributes on facilitating organizations the adoption of data-driven strategies into a cohesive framework operationalizing and automating data governance.

Keywords: Big Data; Data Governance; Data Integration; Metadata.

PubMed Disclaimer

Conflict of interest statement

Competing interestsThe authors have no competing interests to declare that are relevant to the content of this article.

Figures

Fig. 1
Fig. 1
The organization data assets from [4] (in gray those covered by this paper)
Fig. 2
Fig. 2
WISCENTD Use Case
Fig. 3
Fig. 3
High-level overview of the proposed Data Lake architecture and its zones
Fig. 4
Fig. 4
Example of the Temporal Landing for the NTD-related data
Fig. 5
Fig. 5
Persistent landing for medicine request data
Fig. 6
Fig. 6
Example of the schema for the tables in Formatted Zone for the WIMEDS data
Fig. 7
Fig. 7
Example of the medicine request dashboards in the Exploitation Zone
Fig. 8
Fig. 8
UML class diagram for the metadata artifacts
Fig. 9
Fig. 9
Metadata artifacts and their attributes
Fig. 10
Fig. 10
Instantiation of the system architecture for WISCENTD use case
Fig. 11
Fig. 11
Data Collectors DGP
Fig. 12
Fig. 12
Execution of the Data Collectors DGP (WISCENTD Use Case)
Fig. 13
Fig. 13
Data Persistence Loaders DGP
Fig. 14
Fig. 14
Execution of the Data Persistence Loader DGP (WISCENTD Use Case)
Fig. 15
Fig. 15
Data Formatters DGP
Fig. 16
Fig. 16
Execution of the Data Formatters DGP (WISCENTD Use Case)
See this image and copyright information in PMC

References

    1. Horrocks I, Giese M, Kharlamov E, Waaler A. Using semantic technology to tame the data variety challenge. IEEE Internet Comput. 2016;20(6):62–66. doi: 10.1109/MIC.2016.121. - DOI
    1. Popovic A, Hackney R, Tassabehji R, Castelli M. The impact of big data analytics on firms’ high value business performance. Inf Syst Front. 2018;20(2):209–222. doi: 10.1007/s10796-016-9720-4. - DOI
    1. Weill P, Ross JW. IT Governance: How Top Performers Manage IT Decision Rights for Superior Results. New York: Harvard Business Press; 2004.
    1. Khatri V, Brown CV. Designing data governance. Commun ACM. 2010;53(1):148–152. doi: 10.1145/1629175.1629210. - DOI
    1. García S, Romero O, Raventós R. DSS from an RE perspective: a systematic mapping. J Syst Softw. 2016;117:488–507. doi: 10.1016/j.jss.2016.03.046. - DOI

LinkOut - more resources