karpathy/build-nanogpt
Build Nanogpt
This is an educational implementation that builds a generative pre-trained transformer (GPT) language model from scratch using PyTorch. The project is structured as a step-by-step tutorial, walking through the construction of a decoder-only transformer architecture and its training loop with clean git commits and an accompanying video lecture for a hands-on learning experience.
What sets this implementation apart is its focus on practical reproduction: it provides a workflow to train a 124-million-parameter model from scratch in about one hour on cloud GPU hardware, costing under ten dollars. The tutorial covers both the architecture construction and the full training pipeline, making it suitable for those who want to understand the inner workings of a GPT-scale model without relying on pre-built frameworks.
The technical implementation covers the core components of a decoder-only transformer, including causal masked self-attention where each token attends only to preceding tokens, cross-entropy loss minimization for next-token prediction, weight-decay regularization to prevent overfitting, and GPU-accelerated training through PyTorch for large-scale computation. While the project is small in scale, it mirrors the architectural patterns used in larger language models.
Features
- Decoder Architectures - Implements the core decoder-only transformer architecture that processes token sequences for autoregressive generation.
- From-Scratch Decoder Implementations - Focuses on implementing the decoder-only transformer from scratch with causal self-attention and weight-decay.
- GPU-Accelerated Training - Executes forward and backward passes on CUDA-capable GPUs to accelerate large-scale model training.
- Language Model Training - Provides a complete implementation for training a GPT-2-scale language model from scratch with cloud GPU optimization.
- 124M-Parameter Reproduction Guides - Provides a guide for training a 124-million-parameter language model on cloud GPUs for under ten dollars.
- Triangular Mask Implementations - Implements the triangular attention mask that enforces unidirectional token access in the self-attention mechanism.
- Cross-Entropy Loss Functions - Uses cross-entropy loss as the objective function for next-token prediction during language model training.
- Cloud GPU Reproduction Pipelines - Ships a workflow that reproduces a 124M-parameter model on cloud GPU hardware in about one hour for under ten dollars.
- Transformer Language Models - Implements a decoder-only transformer language model that processes token sequences and predicts the next token.
- From-Scratch Training - Demonstrates the complete workflow of training a generative pre-trained transformer from scratch on text data.
- From-Scratch Implementations - Builds a generative pre-trained transformer entirely from scratch, covering both architecture construction and training loop.
- Deep Learning Prototyping Kits - Provides a hands-on environment for prototyping transformer components like causal self-attention and loss functions.
- PyTorch Tensor Operations - Leverages PyTorch for all tensor operations and automatic differentiation throughout the model implementation.
- Transformer Tutorials - Provides a step-by-step walkthrough for implementing a transformer language model using PyTorch.
- Transformer Architecture Walkthroughs - Provides a structured tutorial with clean git commits and video lecture for building a transformer language model.
- Courses and Tutorials - Lecture on building GPT models from scratch.
- Learning Resources - Lecture series on building GPT models.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
