togethercomputer/RedPajama-Data
RedPajama Data
RedPajama-Data es un conjunto de herramientas para el preprocesamiento de conjuntos de datos de texto a gran escala utilizados para entrenar modelos de lenguaje grandes. Proporciona una canalización de preprocesamiento centrada en la limpieza, deduplicación y puntuación de colecciones masivas de texto para garantizar la calidad y diversidad de los datos.
El proyecto utiliza un framework de puntuación de calidad de documentos que emplea aprendizaje automático y heurísticas estadísticas para evaluar si los documentos son adecuados para el entrenamiento. Incluye una canalización de filtrado de conjuntos de datos que utiliza clasificadores y listas de bloqueo para eliminar palabras o URLs no deseadas.
El sistema cuenta con un conjunto de herramientas de deduplicación de texto que elimina contenido redundante utilizando técnicas de coincidencia exacta y difusa. Estas capacidades permiten la identificación y eliminación de documentos duplicados o casi idénticos en un corpus.
Features
- Text Dataset Preparation - Cleans and refines massive text datasets to ensure high quality and diversity for training large language models.
- Dataset Filtering Pipelines - Provides automated systems for cleaning and refining training data through classifiers and blocklists.
- Dataset Preprocessing Tools - Provides a comprehensive toolset for cleaning, deduplicating, and scoring large-scale text datasets for LLM training.
- Training Data Quality Scoring - Uses statistical patterns and machine learning models to assign suitability scores to individual documents for model training.
- Distributed Data Processing - Processes massive web-scale datasets by partitioning document streams across multiple compute nodes for parallel analysis.
- Dataset Deduplication - Identifies and removes near-identical content by calculating similarity signatures instead of relying on exact string matches.
- Large-Scale Deduplications - Removes duplicate or nearly identical documents from massive corpora using distributed computing for high throughput.
- Web Content Deduplication - Eliminates duplicate documents from web-scale datasets using exact content matching and fuzzy matching techniques.
- Document Quality Analysis - Assigns quality scores to documents using machine learning and statistical heuristics to determine suitability for training.
- Topic Classifiers - Provides a pipeline using classifiers and blocklists to filter undesirable content from large-scale datasets.
- Content Signature Computation - Generates quality metrics and unique signatures to identify nearly identical content across a dataset.
- Document Content Hashing - Generates unique fingerprints for documents to detect redundancy and track content across different data sources.
- Text Quality Filtering - Builds classifiers and blocklists of undesirable words or URLs to filter datasets based on quality metrics.
- Large Language Models - Large-scale open-source dataset for training language models.
- Data Resources - Large-scale dataset for pretraining language models.
- Pre-training Datasets - Reproduced dataset for training large-scale language models.
- Large Language Models (LLMs) - Listed in the “Large Language Models (LLMs)” section of the The Incredible Pytorch awesome list.
