Handling missing data when using Goldstein et al.’s Scalelink method of data linkage

Scalelink is an innovative probabilistic data linkage method based on multiple correspondence analysis. Unlike the current gold-standard probabilistic algorithm, Fellegi-Sunter, it does not assume linkage variable independence. All real-world linkage contains dependent linkage variables. Thus, avoiding this assumption has potential to improve data linkage quality. However, to date there are no recommendations regarding handling missing data values when using Scalelink. As all real-world data contains missingness, this means Scalelink cannot currently be used for real-world linkage. Seven candidate methods were identified from the literature as being used in multiple correspondence analysis. They were iteratively tested, first using tiny artificial datasets and subsequently using random samples of real-world census data. These tests were performed using a novel Python/PySpark implementation of Scalelink. Initial testing resulted in a three-method short-list: the Missing Single method (which treats missingness as an agreement state) and two variants of the Missing Insertion method (which removes missingness by imputing donor values). Following testing, results were analysed by compar