togethercomputer/RedPajama-Data

• datajuicer/data-juicer

  datajuicer/data-juicer

  6,574GitHub पर देखें↗

  Data-Juicer is an open-source framework for cleaning, filtering, deduplicating, and transforming multimodal datasets to prepare them for training large language and vision models. It functions as a distributed data pipeline engine that runs processing jobs across Ray clusters, handling billions of samples with automatic operator fusion and adaptive parallelism. The framework provides a library of operators that leverage large language models for semantic extraction, filtering, and data synthesis within processing pipelines. The project distinguishes itself through a YAML-based data recipe sys

  Pythondatadata-analysisdata-pipeline

  GitHub पर देखें↗6,574
• esbatmop/mnbvc

  esbatmop/MNBVC

  4,123GitHub पर देखें↗

  MNBVC is a dataset pipeline and toolkit designed for the collection, cleaning, and normalization of massive text and code corpora used to train large language models. It provides specialized tools for harvesting source code, commit histories, and repository metadata from version control platforms, alongside a multilingual text corpus collector for gathering parallel text and academic papers. The project distinguishes itself through comprehensive capabilities for processing diverse document types, including a PDF-to-text converter that transforms complex layouts and formulas into structured JS

  chinesechinese-languagechinese-nlp

  GitHub पर देखें↗4,123
• eleutherai/gpt-neox

  EleutherAI/gpt-neox

  7,392GitHub पर देखें↗

  gpt-neox is a distributed training system and framework for building large-scale autoregressive language models. It implements the transformer architecture and provides a toolkit for training models with billions of parameters by distributing weights across compute clusters. The framework distinguishes itself through extensive support for distributed model parallelism, including pipeline and sequence parallelism, to overcome single-device memory limits. It further supports sparse model architectures using a mixture of experts system with Sinkhorn-based routing. The project covers a broad ran

  Pythondeepspeed-librarygpt-3language-model

  GitHub पर देखें↗7,392
• facebookresearch/metaseq

  facebookresearch/metaseq

  6,546GitHub पर देखें↗

  Metaseq is a transformer sequence modeling toolkit designed for training, fine-tuning, and deploying sequence-to-sequence models using open pre-trained weights. It provides a comprehensive framework for large language model training, including dedicated tools for sequence dataset processing and a standalone inference server for generating text via API requests. The project features specialized utilities for model quantization to reduce parameter precision to eight bits, which lowers memory usage and increases inference speed. It also includes a checkpoint conversion pipeline to transform mode

  Python

  GitHub पर देखें↗6,546

अक्सर पूछे जाने वाले प्रश्न