EleutherAI/gpt-neox

Features

• Large Language Model Training Frameworks - Provides a distributed framework for training massive autoregressive transformer models across GPU clusters.
• Distributed Training - Manages model, pipeline, and sequence parallelism to distribute large weights across compute clusters.
• Large Language Models - Provides a comprehensive toolkit for building, tokenizing, and aligning large-scale generative text models.
• Large-Scale Model Training - Facilitates training autoregressive models with billions of parameters that exceed single-device memory capacity.
• Model Parallelism - Splits model parameters across multiple GPUs to enable training of models exceeding single device memory.
• Model Parallelism Frameworks - Distributes model layers across multiple GPUs using pipeline, model, and sequence parallelism.
• Pipeline Parallelism Partitioners - Partitions large neural networks into sequential layers across multiple GPUs for pipeline-parallel training.
• Transformer Architecture Implementation - Implements the full transformer architecture with support for massive parameter counts and mixture of experts.
• Training Node Orchestration - Coordinates training workloads across compute clusters using MPI or Slurm for synchronized parallel execution.
• Hardware Acceleration Kernels - Accelerates training through fused-kernel attention mechanisms tailored for specific hardware architectures.
• Hyperparameter Configurations - Enables the definition of model hyperparameters such as layer count, hidden size, and attention head configurations.
• Training Datasets - Retrieves and manages large-scale text collections curated for pre-training generative AI models.
• Learning Rate Decay Schedules - Provides cosine and linear schedules to adjust the learning rate over time for stable convergence.
• Mixed Precision Training - Employs lower-bit precision formats and gradient accumulation to accelerate training and reduce memory consumption.
• Model Performance Benchmarking - Tests pretrained models on standardized benchmarks to quantify accuracy and performance across linguistic tasks.
• Mixture of Experts - Allows configuration of sparse architectures, including expert counts and routing algorithms for MoE layers.
• Sinkhorn Routing - Implements balanced Sinkhorn-based routing to manage token distribution across sparse expert layers.
• Byte Pair Encodings - Implements byte-pair encoding to convert raw text into discrete sequences for model input.
• Attention Kernel Fusion - Combines multiple GPU operations into single kernels to reduce memory bottlenecks and accelerate attention passes.
• RLHF Alignment Algorithms - Implements reinforcement learning algorithms to align model outputs with human preferences.
• Autoregressive Text Generation - Enables token-by-token text generation through pretrained autoregressive models in various interactive modes.
• Sparse Model Architectures - Supports sparse model architectures using a mixture of experts system to improve computational efficiency.
• Text Dataset Preparation - Tokenizes raw text or JSONL files into binary formats for efficient large-scale language model training.
• Text Tokenizers - Provides various tokenizer implementations including BPE, SentencePiece, and Tiktoken to process raw text.
• Dataset Blending - Handles training, validation, and test data paths with support for weighted sampling from multiple sources.
• Training Throughput Optimization - Increases training efficiency through mixed precision, gradient accumulation, and memory optimization techniques.
• Foundation Models - Parallelized autoregressive transformer implementation for GPU clusters.
• Large Language Models - Framework for training large-scale transformer models.
• Model Training Frameworks - Implementation of model-parallel autoregressive transformers on GPUs.
• Pre-trained Language Models - Large-scale open-source autoregressive language model.

Star history