# google-research/deduplicate-text-datasets

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/google-research-deduplicate-text-datasets).**

_How this analysis was created: the description and tags below were written by an AI model that read this project's README and public documentation pages; stars, license and language come straight from the GitHub API. The model does not read the source code._

1,273 stars · 129 forks · Rust · Apache-2.0 · archived

## Links

- GitHub: https://github.com/google-research/deduplicate-text-datasets
- awesome-repositories: https://awesome-repositories.com/repository/google-research-deduplicate-text-datasets.md

## Description

Deduplicate-text-datasets is a data cleaning pipeline and text indexing engine built in Rust for large language model data preparation and corpus decontamination. It constructs massive uncompressed suffix arrays on disk using single-threaded and parallel chunk-based algorithms, enabling memory-efficient pattern searching and indexing across multi-gigabyte text corpora. 

The software identifies overlapping byte ranges from repeated substring matches to strip duplicate content and produce cleaned text datasets. Its search and indexing capabilities support querying corpus occurrences, locating duplicate subsequences that exceed specific length thresholds, and executing cross-dataset content comparisons to find shared overlaps between distinct text corpora.

## Tags

### Artificial Intelligence & ML

- [Data Preparation Pipelines](https://awesome-repositories.com/f/artificial-intelligence-ml/large-language-models/large-language-model-input-tools/data-preparation-pipelines.md) — Cleans and filters massive text corpora to remove duplicated content before training language models.
- [Text Dataset Curators](https://awesome-repositories.com/f/artificial-intelligence-ml/dataset-management/evaluation-datasets/dataset-curation/text-dataset-curators.md) — Scans text documents for overlapping byte ranges and removing repeated subsequences to create cleaner training sets.
- [Dataset Deduplication Tools](https://awesome-repositories.com/f/artificial-intelligence-ml/large-language-models/large-language-model-input-tools/dataset-deduplication-tools.md) — Provides a data cleaning pipeline that builds suffix array indexes to locate and remove repeated text sequences from language model training corpora.

### Data & Databases

- [Corpus Decontamination Utilities](https://awesome-repositories.com/f/data-databases/corpus-decontamination-utilities.md) — Compares text datasets in parallel by walking their suffix arrays to identify overlapping content and cross-document duplicates.
- [Dataset Deduplication](https://awesome-repositories.com/f/data-databases/fuzzy-matching/dataset-deduplication.md) — Identifies overlapping byte ranges from repeated substring matches to strip duplicate content and produce cleaned text datasets. ([source](https://github.com/google-research/deduplicate-text-datasets/blob/master/README.md))
- [Cross-Dataset Content Comparisons](https://awesome-repositories.com/f/data-databases/cross-dataset-content-comparisons.md) — Compares two separate text corpora in parallel using suffix arrays to find shared content and overlaps between them.

### Software Engineering & Architecture

- [Suffix Array Construction](https://awesome-repositories.com/f/software-engineering-architecture/string-processing-algorithms/suffix-array-construction.md) — Constructs massive uncompressed suffix arrays on disk to enable memory-efficient pattern searching and indexing across multi-gigabyte text corpora.
- [Suffix Array Indexing Engines](https://awesome-repositories.com/f/software-engineering-architecture/string-processing-algorithms/suffix-array-construction/suffix-array-indexing-engines.md) — Constructs suffix arrays for large text files using single-threaded and parallel chunk-based algorithms for fast substring lookups.
- [Cross-Document Duplicate Detectors](https://awesome-repositories.com/f/software-engineering-architecture/in-place-array-manipulations/duplicate-limiting-in-sorted-arrays/duplicate-detection-in-arrays/cross-document-duplicate-detectors.md) — Compares distinct text datasets by walking their suffix arrays in parallel to identify overlapping content. ([source](https://github.com/google-research/deduplicate-text-datasets/blob/master/README.md))
- [Repeated Substring Detection](https://awesome-repositories.com/f/software-engineering-architecture/substring-search-algorithms/repeated-substring-detection.md) — Scans datasets to locate substrings exceeding specific length thresholds that appear multiple times across documents. ([source](https://github.com/google-research/deduplicate-text-datasets#readme))