EleutherAI/gpt-neoArchived
Gpt Neo
GPT-Neo is an open-source distributed training framework designed for scaling GPT-2 and GPT-3-style language models across multiple devices using mesh-tensorflow for model parallelism. It provides the infrastructure to train transformer-based language models with billions of parameters across distributed computing environments, making large-scale language model research accessible outside of proprietary systems.
The framework supports training both autoregressive GPT-style models and masked language models like BERT or RoBERTa, with configurable masking strategies and token handling. It includes capabilities for fine-tuning models through reinforcement learning from human feedback, enabling alignment of model outputs with human preferences. For evaluation, GPT-Neo provides standardized benchmarking tools with contamination detection to ensure reproducible and transparent assessment of language model performance.
Beyond training and evaluation, the project encompasses interpretability research tools for analyzing internal representations across transformer layers, including techniques for behavior attribution, concept erasure, and latent knowledge elicitation. It also supports multimodal data processing to extend language model research into image and audio domains. The framework implements memory-efficient training techniques such as gradient checkpointing, mixed-precision arithmetic, and dynamic batching to maximize hardware utilization during large-scale training runs.
Features
- Mesh-Tensorflow Parallelism Implementations - Provides the core mesh-tensorflow infrastructure for distributing large transformer computations across multiple devices.
- Large Language Model Training Frameworks - Provides a distributed training framework for scaling GPT-style language models across multiple devices using mesh-tensorflow.
- Autoregressive Transformers - Ships an autoregressive transformer decoder that generates text token-by-token with causal attention masking.
- Autoregressive Text Generation - Generates text continuations from prompts using trained or pre-trained transformer models with a prediction flag.
- Inference Execution - Generates text continuations from trained transformer models by feeding prompts through the network.
- Model Inference - Generates text continuations from trained transformer models by feeding prompts through the network.
- Human Preference Alignment - Implements reinforcement learning from human feedback to fine-tune language models and align outputs with preferences.
- Mesh-Tensorflow Parallelism Implementations - Implements model parallelism using mesh-tensorflow to distribute large neural network computations across many devices.
- Distributed Layer Sharders - Splits transformer layers across GPU/TPU devices to enable training of models with tens of billions of parameters.
- Reward Model Aligners - Implements reinforcement learning from human feedback to align model outputs with intended human goals.
- Decision Process Analyzers - Provides tools for analyzing internal model decisions to predict and modify AI behavior for safer deployment.
- Batch Sequence Training - Groups variable-length sequences into padded batches to maximize hardware utilization during training.
- Gradient Checkpointing - Implements gradient checkpointing to trade compute for memory by recomputing intermediate activations during backpropagation.
- Concept Erasure Techniques - Implements techniques for removing specific linear concepts from model representations in closed form.
- Configurable Masking Strategies - Supports configurable masking probabilities and patterns for training BERT and RoBERTa-style models.
- Reproducible Evaluation Harnesses - Provides standardized benchmarking tools with contamination detection for reproducible language model evaluation.
- Mixed Precision Training - Uses float16 or bfloat16 arithmetic for forward and backward passes while maintaining float32 master weights for stability.
- Masked Language Modeling - Supports training models like BERT and RoBERTa using a masked-language-modeling objective with configurable masking probabilities.
- Training Pipelines - Supports training BERT-style models with configurable masking strategies and token handling for masked language modeling.
- Behavior Elicitation Techniques - Provides tools for probing and triggering targeted behaviors in language models for interpretability research.
- Sparse Autoencoder Attributors - Identifies which internal features of a language model cause its outputs using sparse autoencoders.
- Latent Knowledge Elicitation Techniques - Provides techniques for extracting knowledge from internal model representations even when outputs are deceptive.
- Transparent Model Suites - Provides a suite of models trained with transparent methods to enable reproducible scientific studies of language model behavior.
- Standardized Evaluation Harnesses - Provides a standardized evaluation harness for reproducible and transparent language model benchmarking.
- Transformer Representation Analysis - Ships tools for extracting and studying internal representations across transformer layers for interpretability.
- Transformer Representation Analyzers - Ships tools for extracting and analyzing internal representations across transformer layers for interpretability research.
- Dataset Contamination Detection - Quantifies how pretraining data contamination affects generative model outputs by sweeping model sizes and test set replicas.
