Metadata-Driven Approach to Generalisation of Transformations in ETL Processes

Abstract

Introduction: The secondary use of clinical data becomes more important, whereby a large number of ETL routes for data integration are implemented for specific purposes. The metadata driven approach allows a generalization of ETL processes to reuse existing implementations.

Methods: The metadata stored in the MDR, which describes different attributes of data, is used for this purpose, based on the ISO 21526 and governance and provenance data were taken into account to record the history of the data throughout its lifecycle, based on the W3C PROV model.

Results: To achieve a metadata-driven approach the data structure of the source and target system are represented in a MDR. Afterwards, relations between elements have to be defined with respective transformation rules. These information are used by an generic ETL implementation, so that use case specific content is outsourced within the MDR.

Discussion: A rule-based approach of the MDD ETL implementation allows a generalization of the extract and load phase, however the transformation process has to be standardized further. Moreover, a user-friendly interface is essential for integrating expertise without technical skills.

Keywords: Data Integration; ETL process; MDR; Metadata-driven approach; OMOP CDM; metadata; rule-based approach; secondary use.