Data Integration for Heterogenous Datasets

Abstract

More and more, the needs of data analysts are requiring the use of data outside the control of their own organizations. The increasing amount of data available on the Web, the new technologies for linking data across datasets, and the increasing need to integrate structured and unstructured data are all driving this trend. In this article, we provide a technical overview of the emerging "broad data" area, in which the variety of heterogeneous data being used, rather than the scale of the data being analyzed, is the limiting factor in data analysis efforts. The article explores some of the emerging themes in data discovery, data integration, linked data, and the combination of structured and unstructured data.

FIG. 1.

Users increasingly need to integrate data from multiple sources, and of multiple types, into their strategic data analyses.

FIG. 2.

A simple example integrating U.S. and Chinese data with an intermediary dataset.

FIG. 3.

The U.S. Data.gov and many other government web sites provide faceted browsers for searching through large dataset catalogs based on metadata features.

FIG. 4.

Comparing U.S. foreign aid. (a) Comparing U.S. foreign aid from various agencies. Data is reconciled by country, amounts, and years. (b) Extending the comparison to show U.S. foreign aid from various agencies and the specific types of aid over time. (c) Comparing U.S. foreign aid and British foreign aid by combining datasets and using the currency conversion data show in Figure 2.

FIG. 5.

Database linking example.

FIG. 6.

A tabular depiction of some of the different ways the state “Alaska” is referred to in different databases.

FIG. 7.

A comparison of burglary data between a U.K. and a U.S. dataset.

FIG. 8.

News titles, tweets with sentiment, and mapping use time and geolocation features to correlate and display information.