How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
This project is a comprehensive, community-maintained knowledge base and toolkit designed for competitive programming. It serves as a centralized repository for algorithmic theory, data structures, and mathematical techniques, providing a structured reference for informatics and collegiate programming competitions. The project distinguishes itself by integrating educational content with a robust suite of automation utilities. It provides a complete workflow for competitive programming, including tools for automated test case generation, solution verification, and direct interaction with onlin
Data-Juicer is an open-source framework for cleaning, filtering, deduplicating, and transforming multimodal datasets to prepare them for training large language and vision models. It functions as a distributed data pipeline engine that runs processing jobs across Ray clusters, handling billions of samples with automatic operator fusion and adaptive parallelism. The framework provides a library of operators that leverage large language models for semantic extraction, filtering, and data synthesis within processing pipelines. The project distinguishes itself through a YAML-based data recipe sys
Open-Instruct is a distributed training and instruction tuning framework for large language models. It functions as a coordinator for supervised fine-tuning, reinforcement learning from human feedback pipelines, and tool-use training, providing specialized roles for dataset curation and model alignment. The project distinguishes itself through a high-performance training architecture that utilizes actor-based distributed coordination and hybrid sharding to manage large GPU clusters. It implements advanced alignment techniques including direct preference optimization, group relative policy opt
Deduplicate-text-datasets is a data cleaning pipeline and text indexing engine built in Rust for large language model data preparation and corpus decontamination. It constructs massive uncompressed suffix arrays on disk using single-threaded and parallel chunk-based algorithms, enabling memory-efficient pattern searching and indexing across multi-gigabyte text corpora.
The main features of google-research/deduplicate-text-datasets are: Data Preparation Pipelines, Text Dataset Curators, Dataset Deduplication Tools, Corpus Decontamination Utilities, Dataset Deduplication, Suffix Array Construction, Suffix Array Indexing Engines, Cross-Dataset Content Comparisons.
Projects with overlapping indexed features include: oi-wiki/oi-wiki — This project is a comprehensive, community-maintained knowledge base and toolkit designed for competitive programming.… opendcai/dataflow — DataFlow is an agent-based workflow orchestrator and data pipeline designed to synthesize, clean, and augment… allenai/open-instruct — Open-Instruct is a distributed training and instruction tuning framework for large language models. It functions as a… nvidia/isaac-gr00t. datajuicer/data-juicer — Data-Juicer is an open-source framework for cleaning, filtering, deduplicating, and transforming multimodal datasets… togethercomputer/redpajama-data — RedPajama-Data is a toolset for preprocessing large-scale text datasets used to train large language models. It…
