Developing an Evaluation Toolkit for Data Linkage Pipelines

ObjectivesThe linkage evaluation toolkit aims to develop criteria that can be used to evaluate data linkage pipelines and methods. This toolkit will not only allow assessment of a single pipeline, but will also facilitate the comparison between additional pipelines; enabling analysts to identify their strengths and weaknesses. MethodsThere is currently little information available on standardised evaluation metrics for linkage pipelines, except the quality of the resulting linked dataset. Relevant teams across the organisation were consulted in a series of workshops to identify existing methods of evaluating pipelines across 8 key areas; accuracy, bias in linkage error, flexibility in input datasets, scalability, efficiency, platform suitability, ease of use, and the ease and transparency of quality assurance. Existing methods, tools, and criteria for evaluating each area were discussed, and where these did not exist, discussion aimed to identify potential solutions. Results Evaluation criteria for each area were identified. Whilst some areas had quantitative metrics such as precision and recall, other areas had more qualitative criteria, such as clarity of the methods documentat