togethercomputer/RedPajama-Data
RedPajama Data
RedPajama-Data is a toolset for preprocessing large-scale text datasets used to train large language models. It provides a preprocessing pipeline focused on cleaning, deduplicating, and scoring massive collections of text to ensure data quality and diversity.
The project utilizes a document quality scoring framework that employs machine learning and statistical heuristics to evaluate if documents are suitable for training. It includes a dataset filtering pipeline that uses classifiers and blocklists to remove undesirable words or URLs.
The system features a text deduplication toolset that eliminates redundant content using both exact and fuzzy matching techniques. These capabilities allow for the identification and removal of duplicate or nearly identical documents across a corpus.
Features
- Text Dataset Preparation - Cleans and refines massive text datasets to ensure high quality and diversity for training large language models.
- Dataset Filtering Pipelines - Provides automated systems for cleaning and refining training data through classifiers and blocklists.
- Dataset Preprocessing Tools - Provides a comprehensive toolset for cleaning, deduplicating, and scoring large-scale text datasets for LLM training.
- Training Data Quality Scoring - Uses statistical patterns and machine learning models to assign suitability scores to individual documents for model training.
- Distributed Data Processing - Processes massive web-scale datasets by partitioning document streams across multiple compute nodes for parallel analysis.
- Dataset Deduplication - Identifies and removes near-identical content by calculating similarity signatures instead of relying on exact string matches.
- Large-Scale Deduplications - Removes duplicate or nearly identical documents from massive corpora using distributed computing for high throughput.
- Web Content Deduplication - Eliminates duplicate documents from web-scale datasets using exact content matching and fuzzy matching techniques.
- Document Quality Analysis - Assigns quality scores to documents using machine learning and statistical heuristics to determine suitability for training.
- Topic Classifiers - Provides a pipeline using classifiers and blocklists to filter undesirable content from large-scale datasets.
- Content Signature Computation - Generates quality metrics and unique signatures to identify nearly identical content across a dataset.
- Document Content Hashing - Generates unique fingerprints for documents to detect redundancy and track content across different data sources.
- Text Quality Filtering - Builds classifiers and blocklists of undesirable words or URLs to filter datasets based on quality metrics.
- Large Language Models - Large-scale open-source dataset for training language models.
- Data Resources - Large-scale dataset for pretraining language models.
- Pre-training Datasets - Reproduced dataset for training large-scale language models.
- Large Language Models (LLMs) - Listed in the “Large Language Models (LLMs)” section of the The Incredible Pytorch awesome list.
