awesome-repositories.com
Blog
MCP
awesome-repositories.com

Descoperă cele mai bune repository-uri open source cu căutare AI.

ExploreazăCăutări recomandateAlternative open-sourceSoftware self-hostedBlogHartă site
ProiectDespreCum realizăm clasamentulPresăServer MCP
LegalConfidențialitateTermeni
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
·
dedupeio avatar

dedupeio/dedupe

0
View on GitHub↗
docs.dedupe.io
↗

Dedupe

Dedupe is a machine learning tool for entity resolution that identifies and merges duplicate records in structured datasets. It uses active learning to train a matching model from human-labeled examples, learning which field-level similarities are most important for detecting duplicates without requiring manual rule writing. The system combines fingerprint-based blocking to reduce pairwise comparisons, enabling efficient matching on large datasets, and groups scored record pairs into clusters using a configurable similarity threshold.

The tool provides multiple interfaces for different workflows. A command-line tool allows deduplicating or linking records in CSV files without writing any Python code, while a web-based service supports uploading data, training models, and reviewing results without local setup. Trained matching configurations and blocking rules can be saved to files for reuse across sessions without retraining. The system also supports cross-dataset record linking, matching records from separate data sources that refer to the same entity without requiring a shared unique identifier.

Beyond core deduplication, the tool offers capabilities for constructing canonical records from clusters by selecting the most common value for each field, matching messy records against a clean reference dataset, and generating training data from already deduplicated datasets. Custom comparators, data types, and blocking rules can be added for domain-specific matching, and the system is designed to perform intelligent record comparisons that scale to large datasets without requiring a powerful server.

Features

  • Entity Deduplication Tools - Ships a machine learning tool that identifies and merges duplicate records in structured datasets using active learning.
  • Active Learning Training Workflows - Trains a machine learning model from human-labeled examples to identify and merge duplicate records in structured datasets.
  • Entity Resolution - Merges or links matched records into a single canonical representation, removing redundancy from a dataset.
  • Entity Clustering and Canonicalization - Uses learned similarity metrics to compare record pairs, group them into clusters, and construct canonical records.

Căutare AI

Explorează mai multe repository-uri excelente

Descrie ce ai nevoie în limbaj simplu — AI-ul sortează mii de proiecte open source selectate în funcție de relevanță.

Start searching with AI
4,442 stele·570 fork-uri·Python·mit·9 vizualizări
  • Custom Deduplication Rule Trainers - Learns optimal matching weights and blocking rules from human-labeled example pairs.
  • Matching Model Optimization - Presents the most uncertain record pairs for a user to label as match or distinct, incrementally improving the model.
  • Entity Resolution Pair Scorers - Computes a probability score for each pair of records indicating how likely they are to refer to the same entity.
  • Iterative Human-in-the-Loop Labeling - Presents uncertain record pairs in the console for user labeling to refine the matching model.
  • Blocking-Based Record Matchers - Reduces the number of record comparisons in large datasets by using fingerprint-based blocking to improve performance.
  • Fuzzy Matching - Compares pairs of records using learned fuzzy rules and groups those that likely refer to the same real-world entity.
  • Machine Learning Deduplicators - Uses machine learning to identify and merge duplicate entries in structured data based on learned rules.
  • Inter-Dataset Record Linkage - Matches records from separate data sources that refer to the same real-world entity without shared unique identifiers.
  • Intra-Dataset Deduplication - Identifies records that refer to the same entity within one dataset and groups them into clusters with confidence scores.
  • Cross-Dataset Record Matchers - Finds pairs of records that refer to the same entity across two separate datasets, supporting one-to-one, many-to-one, and many-to-many constraints.
  • Duplicate Record Groupers - Identifies and groups records across a dataset that likely represent the same entity based on learned matching rules.
  • Record Clustering - Groups scored record pairs into clusters of records that all refer to the same entity.
  • CSV Deduplication Utilities - Runs a command-line tool that deduplicates or links records in CSV files without writing Python code.
  • Active Learning Matchers - Trains matching rules from human-labeled examples to automate duplicate detection across datasets.
  • Field-Level Similarity Learners - Learns from labeled examples which field-level similarities are most important for identifying duplicates in a specific dataset.
  • Learned Similarity Metrics - Provides trainable similarity metrics that learn field-level weights from labeled examples for entity matching.
  • Fuzzy Entity Deduplicators - Identifies and merges entries that refer to the same real-world entity, even when names or addresses differ slightly.
  • Model Persistence - Saves trained matching configurations and blocking rules to a settings file for consistent reuse across sessions.
  • Blocking-Based Pair Generators - Yields pairs of records that share blocking fingerprints, reducing the number of comparisons needed for large datasets.
  • Training Data Generation - Builds labeled training examples from an already deduplicated or linked dataset that has a common key identifying matches.
  • Terminal-Based Labeling Interfaces - Presents uncertain record pairs in the terminal for user labeling to incrementally improve the matching model.
  • Entity Resolution Model Checkpoints - Restores a previously trained matching model from a settings file for prediction without retraining.
  • Canonical Data Matching - Indexes a clean reference dataset and searches messy records against it to find the best matching canonical records.
  • Canonical Record Generation - Creates a single representative record from a cluster of duplicates by selecting the most common field values.
  • Canonical Record Synthesis - Creates a single representative record from a cluster by selecting the most common value for each field.
  • Laptop-Scale Deduplication - Performs intelligent record comparisons that scale to large datasets without requiring a powerful server.
  • Custom Data Comparators - Supports adding custom data types, string comparators, and blocking rules for domain-specific matching.
  • Natural Language Processing - Library for fuzzy matching and record linkage.

    • Istoric stele

    Graficul istoricului de stele pentru dedupeio/dedupeGraficul istoricului de stele pentru dedupeio/dedupe

    Întrebări frecvente

    Ce face dedupeio/dedupe?

    Dedupe is a machine learning tool for entity resolution that identifies and merges duplicate records in structured datasets. It uses active learning to train a matching model from human-labeled examples, learning which field-level similarities are most important for detecting duplicates without requiring manual rule writing. The system combines fingerprint-based blocking to reduce pairwise comparisons, enabling efficient matching on large datasets, and groups scored record…

    Care sunt principalele funcționalități ale dedupeio/dedupe?

    Principalele funcționalități ale dedupeio/dedupe sunt: Entity Deduplication Tools, Active Learning Training Workflows, Entity Resolution, Entity Clustering and Canonicalization, Custom Deduplication Rule Trainers, Matching Model Optimization, Entity Resolution Pair Scorers, Iterative Human-in-the-Loop Labeling.

    Care sunt câteva alternative open-source pentru dedupeio/dedupe?

    Alternativele open-source pentru dedupeio/dedupe includ: rapidfuzz/rapidfuzz — RapidFuzz is a C++ accelerated Python library providing high-performance string comparison and similarity… ucbepic/docetl — docetl is an AI-powered document ETL tool and map-reduce orchestrator designed to transform large collections of… openvenues/libpostal — Libpostal is a C library designed for international address parsing and normalization. It utilizes statistical NLP and… seatgeek/fuzzywuzzy — Fuzzywuzzy is a Python library and text processing utility designed to calculate similarity scores between strings. It… falkordb/falkordb — FalkorDB is a high-performance graph database management system and vector graph database. It serves as a knowledge… huggingface/sentence-transformers — This project is a transformer-based framework for generating dense and sparse vector embeddings of text and multimodal…

    Alternative open-source pentru Dedupe

    Proiecte open-source similare, clasificate după numărul de funcționalități comune cu Dedupe.
    • rapidfuzz/rapidfuzzAvatar rapidfuzz

      rapidfuzz/RapidFuzz

      3,731Vezi pe GitHub↗

      RapidFuzz is a C++ accelerated Python library providing high-performance string comparison and similarity calculations. It functions as a fuzzy string matching toolkit used to quantify the difference between text sequences through Levenshtein distance and other edit distance metrics. The library focuses on scalable approximate text matching, enabling the identification and ranking of similar strings within large datasets. It provides specialized utilities for finding the best matches in a collection and generating pairwise similarity matrices. The project covers a broad surface of text proce

      Pythoncpplevenshteinlevenshtein-distance
      Vezi pe GitHub↗3,731
    • ucbepic/docetlAvatar ucbepic

      ucbepic/docetl

      3,597Vezi pe GitHub↗

      docetl is an AI-powered document ETL tool and map-reduce orchestrator designed to transform large collections of unstructured documents into structured, queryable tables using language models. It provides a declarative pipeline framework for extracting, cleaning, and transforming data from sources such as PDFs and text files into predefined schemas. The project distinguishes itself through a semantic data integration suite that enables joining datasets and resolving duplicate entities based on embedding-based similarity. It includes an interactive prompt playground for developing and optimizi

      Pythonagentsdatadata-pipelines
      Vezi pe GitHub↗3,597
    • openvenues/libpostalAvatar openvenues

      openvenues/libpostal

      4,819Vezi pe GitHub↗

      Libpostal is a C library designed for international address parsing and normalization. It utilizes statistical NLP and a language classifier to decompose unstructured global address strings into structured components and standardize street addresses by expanding abbreviations and resolving regional naming variations across multiple languages. The project provides tools for text transliteration, converting various scripts into standardized Latin-ASCII or NFD forms. It also includes capabilities for address deduplication, using symmetric fuzzy matching to identify whether different address reco

      C
      Vezi pe GitHub↗4,819
    • seatgeek/fuzzywuzzyAvatar seatgeek

      seatgeek/fuzzywuzzy

      9,258Vezi pe GitHub↗

      Fuzzywuzzy is a Python library and text processing utility designed to calculate similarity scores between strings. It functions as a text similarity scoring engine and an approximate string matching tool used to identify the closest textual matches within a list of candidate strings. The library provides a suite of tools for measuring the degree of similarity between pieces of text, accounting for typos and formatting differences. These capabilities include extracting the best match from a candidate list and performing fuzzy string matching through various scoring methods. The toolset cover

      Python
      Vezi pe GitHub↗9,258
    Vezi toate cele 30 alternative pentru Dedupe→