Probabilistic record linkage

Abstract

Studies involving the use of probabilistic record linkage are becoming increasingly common. However, the methods underpinning probabilistic record linkage are not widely taught or understood, and therefore these studies can appear to be a 'black box' research tool. In this article, we aim to describe the process of probabilistic record linkage through a simple exemplar. We first introduce the concept of deterministic linkage and contrast this with probabilistic linkage. We illustrate each step of the process using a simple exemplar and describe the data structure required to perform a probabilistic linkage. We describe the process of calculating and interpreting matched weights and how to convert matched weights into posterior probabilities of a match using Bayes theorem. We conclude this article with a brief discussion of some of the computational demands of record linkage, how you might assess the quality of your linkage algorithm, and how epidemiologists can maximize the value of their record-linked research using robust record linkage methods.

Keywords: Record linkage; bias; data linkage; epidemiological methods; medical record linkage.

Figure 1

Illustration of two distinct files containing data on sex and education qualification. M, male; F, female; edu, education.

Figure 2

Results of merging two files using first and last name.

Figure 3

Results of joining two files and calculating simple agreement patterns. Fname, first name; Lname, last name; Ag pat, agreement pattern.

Figure 4

Results of joining two files and calculating complex agreement patterns using the edit distance between first name fields in the master file and the file of interest. Fname, first name; Lname, last name; Comp ag pat, complex agreement pattern.

Figure 5

Partitioning of two files into matched and unmatched records.

Figure 6

Matrix representation of true match status of two linked files (File_Master and File_FOI) containing varying frequencies of surname ${\mathbf{M}}_{\gamma =1}$ indicates a true matched pair of records where linkage fields agree (bold on the diagonal), ${\text{M}}_{\gamma =0}$ indicates a true matched pair of records where linkage fields disagree, ${\text{U}}_{\gamma =1}$ indicates a true unmatched pair of records where linkage fields agree (the off diagonal elements in the upper left and lower right quadrants), ${\text{U}}_{\gamma =0}$ indicates a true unmatched pair of records where linkage fields disagree (the lower left and upper right quadrants).

Figure 7

Comparison of results from a diagnostic test against the true disease status and a record linkage against the true match status. Dis, disease.

Figure 8

Calculation of simple agreement weights, ${\text{\hspace{0.17em}log}}_2\text{R}\left({\gamma }_{\text{j}}\right)$, using the Fellegi and Sunter record linkage framework. Fname, first name; Lname, last name; Ag pat, agreement pattern.

Figure 9

Calculating of complex agreement weights, ${\text{\hspace{0.17em}log}}_2\text{R}\left({\gamma }_{\text{j}}^{\text{′}}\right)$, using the Fellegi and Sunter record linkage framework. Fname, first name; Lname, last name; Ag pat, agreement pattern.

Figure 10

Probabilities of links based complex agreement weights, ${\text{\hspace{0.17em}log}}_2\left(\text{R}\left({\gamma }_{\text{j}}^{\text{′}}\right)\right)$ calculated using Bayes theorem. Fname, first name; Lname, last name; Comp ag pat, complex agreement pattern.