Splink – Fast, accurate and scalable probabilistic data linkage
10 hours ago
- Splink is a Python package for probabilistic record linkage (entity resolution) that deduplicates and links records without unique identifiers.
- Key features include high speed (linking ~1 million records on a laptop in about a minute), accuracy with term frequency adjustments and fuzzy matching, and scalability via DuckDB, AWS Athena, or Spark for 100+ million records.
- It uses an unsupervised learning approach, requiring no training data, and provides interactive outputs for model understanding and diagnosis.
- The core algorithm is based on the Fellegi-Sunter model with customizations to improve accuracy.
- Best results are achieved with standardized, multi-column input data that are not highly correlated; it is not designed for single-column 'bag of words' data.
- Splink has been adopted by numerous organizations worldwide, including government agencies, healthcare institutions, and research groups, for tasks such as census linkage, business registers, and health record deduplication.
- It has received multiple awards, including the Civil Service Awards 2025 (Innovation and Excellence in Delivery) and OpenUK Awards 2025.