dedupeio/dedupe
Dedupe
Dedupe is a machine learning tool for entity resolution that identifies and merges duplicate records in structured datasets. It uses active learning to train a matching model from human-labeled examples, learning which field-level similarities are most important for detecting duplicates without requiring manual rule writing. The system combines fingerprint-based blocking to reduce pairwise comparisons, enabling efficient matching on large datasets, and groups scored record pairs into clusters using a configurable similarity threshold.
The tool provides multiple interfaces for different workflows. A command-line tool allows deduplicating or linking records in CSV files without writing any Python code, while a web-based service supports uploading data, training models, and reviewing results without local setup. Trained matching configurations and blocking rules can be saved to files for reuse across sessions without retraining. The system also supports cross-dataset record linking, matching records from separate data sources that refer to the same entity without requiring a shared unique identifier.
Beyond core deduplication, the tool offers capabilities for constructing canonical records from clusters by selecting the most common value for each field, matching messy records against a clean reference dataset, and generating training data from already deduplicated datasets. Custom comparators, data types, and blocking rules can be added for domain-specific matching, and the system is designed to perform intelligent record comparisons that scale to large datasets without requiring a powerful server.
Features
- Entity Deduplication Tools - Ships a machine learning tool that identifies and merges duplicate records in structured datasets using active learning.
- Active Learning Training Workflows - Trains a machine learning model from human-labeled examples to identify and merge duplicate records in structured datasets.
- Entity Resolution - Merges or links matched records into a single canonical representation, removing redundancy from a dataset.
- Entity Clustering and Canonicalization - Uses learned similarity metrics to compare record pairs, group them into clusters, and construct canonical records.
- Custom Deduplication Rule Trainers - Learns optimal matching weights and blocking rules from human-labeled example pairs.
- Matching Model Optimization - Presents the most uncertain record pairs for a user to label as match or distinct, incrementally improving the model.
- Entity Resolution Pair Scorers - Computes a probability score for each pair of records indicating how likely they are to refer to the same entity.
- Iterative Human-in-the-Loop Labeling - Presents uncertain record pairs in the console for user labeling to refine the matching model.
- Blocking-Based Record Matchers - Reduces the number of record comparisons in large datasets by using fingerprint-based blocking to improve performance.
- Fuzzy Matching - Compares pairs of records using learned fuzzy rules and groups those that likely refer to the same real-world entity.
- Machine Learning Deduplicators - Uses machine learning to identify and merge duplicate entries in structured data based on learned rules.
- Inter-Dataset Record Linkage - Matches records from separate data sources that refer to the same real-world entity without shared unique identifiers.
- Intra-Dataset Deduplication - Identifies records that refer to the same entity within one dataset and groups them into clusters with confidence scores.
- Cross-Dataset Record Matchers - Finds pairs of records that refer to the same entity across two separate datasets, supporting one-to-one, many-to-one, and many-to-many constraints.
- Duplicate Record Groupers - Identifies and groups records across a dataset that likely represent the same entity based on learned matching rules.
- Record Clustering - Groups scored record pairs into clusters of records that all refer to the same entity.
- CSV Deduplication Utilities - Runs a command-line tool that deduplicates or links records in CSV files without writing Python code.
- Active Learning Matchers - Trains matching rules from human-labeled examples to automate duplicate detection across datasets.
- Field-Level Similarity Learners - Learns from labeled examples which field-level similarities are most important for identifying duplicates in a specific dataset.
- Learned Similarity Metrics - Provides trainable similarity metrics that learn field-level weights from labeled examples for entity matching.
- Fuzzy Entity Deduplicators - Identifies and merges entries that refer to the same real-world entity, even when names or addresses differ slightly.
- Model Persistence - Saves trained matching configurations and blocking rules to a settings file for consistent reuse across sessions.
- Blocking-Based Pair Generators - Yields pairs of records that share blocking fingerprints, reducing the number of comparisons needed for large datasets.
- Training Data Generation - Builds labeled training examples from an already deduplicated or linked dataset that has a common key identifying matches.
- Terminal-Based Labeling Interfaces - Presents uncertain record pairs in the terminal for user labeling to incrementally improve the matching model.
- Entity Resolution Model Checkpoints - Restores a previously trained matching model from a settings file for prediction without retraining.
- Canonical Data Matching - Indexes a clean reference dataset and searches messy records against it to find the best matching canonical records.
- Canonical Record Generation - Creates a single representative record from a cluster of duplicates by selecting the most common field values.
- Canonical Record Synthesis - Creates a single representative record from a cluster by selecting the most common value for each field.
- Laptop-Scale Deduplication - Performs intelligent record comparisons that scale to large datasets without requiring a powerful server.
- Custom Data Comparators - Supports adding custom data types, string comparators, and blocking rules for domain-specific matching.
- Natural Language Processing - Library for fuzzy matching and record linkage.
