togethercomputer/RedPajama-Data
RedPajama Data
RedPajama-Data ist ein Toolset für das Preprocessing großskaliger Textdatensätze, die zum Training großer Sprachmodelle verwendet werden. Es bietet eine Preprocessing-Pipeline, die sich auf das Bereinigen, Deduplizieren und Bewerten massiver Textsammlungen konzentriert, um Datenqualität und -vielfalt sicherzustellen.
Das Projekt nutzt ein Framework zur Bewertung der Dokumentqualität, das Machine Learning und statistische Heuristiken einsetzt, um zu bewerten, ob Dokumente für das Training geeignet sind. Es enthält eine Datensatz-Filter-Pipeline, die Klassifikatoren und Blocklisten verwendet, um unerwünschte Wörter oder URLs zu entfernen.
Das System verfügt über ein Text-Deduplizierungstoolset, das redundante Inhalte sowohl mit exakten als auch mit Fuzzy-Matching-Techniken eliminiert. Diese Funktionen ermöglichen die Identifizierung und Entfernung doppelter oder nahezu identischer Dokumente innerhalb eines Korpus.
Features
- Text Dataset Preparation - Cleans and refines massive text datasets to ensure high quality and diversity for training large language models.
- Dataset Filtering Pipelines - Provides automated systems for cleaning and refining training data through classifiers and blocklists.
- Dataset Preprocessing Tools - Provides a comprehensive toolset for cleaning, deduplicating, and scoring large-scale text datasets for LLM training.
- Training Data Quality Scoring - Uses statistical patterns and machine learning models to assign suitability scores to individual documents for model training.
- Distributed Data Processing - Processes massive web-scale datasets by partitioning document streams across multiple compute nodes for parallel analysis.
- Dataset Deduplication - Identifies and removes near-identical content by calculating similarity signatures instead of relying on exact string matches.
- Large-Scale Deduplications - Removes duplicate or nearly identical documents from massive corpora using distributed computing for high throughput.
- Web Content Deduplication - Eliminates duplicate documents from web-scale datasets using exact content matching and fuzzy matching techniques.
- Document Quality Analysis - Assigns quality scores to documents using machine learning and statistical heuristics to determine suitability for training.
- Topic Classifiers - Provides a pipeline using classifiers and blocklists to filter undesirable content from large-scale datasets.
- Content Signature Computation - Generates quality metrics and unique signatures to identify nearly identical content across a dataset.
- Document Content Hashing - Generates unique fingerprints for documents to detect redundancy and track content across different data sources.
- Text Quality Filtering - Builds classifiers and blocklists of undesirable words or URLs to filter datasets based on quality metrics.
- Large Language Models - Large-scale open-source dataset for training language models.
- Data Resources - Large-scale dataset for pretraining language models.
- Pre-training Datasets - Reproduced dataset for training large-scale language models.
- Large Language Models (LLMs) - Listed in the “Large Language Models (LLMs)” section of the The Incredible Pytorch awesome list.
