# karpathy/ng-video-lecture

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/karpathy-ng-video-lecture).**

_How this analysis was created: the description and tags below were written by an AI model that read this project's README and public documentation pages; stars, license and language come straight from the GitHub API. The model does not read the source code._

4,798 stars · 1,317 forks · Python

## Links

- GitHub: https://github.com/karpathy/ng-video-lecture
- awesome-repositories: https://awesome-repositories.com/repository/karpathy-ng-video-lecture.md

## Description

This project is an educational implementation of a small-scale generative pre-trained transformer designed to teach the fundamentals of neural network architecture and training. It serves as a reference implementation and tutorial for constructing a text-generating neural network from scratch.

The codebase demonstrates the mechanics of tokenization, self-attention, and the construction of a lightweight language model. It focuses on the step-by-step process of building a generative model to illustrate how large language models are constructed.

The implementation covers transformer-based architecture, including multi-head attention, feed-forward networks, and causal masking. It utilizes PyTorch for tensor computation and employs backpropagation-based learning to train the model on sequential text data.

## Tags

### Artificial Intelligence & ML

- [Transformer Architectures](https://awesome-repositories.com/f/artificial-intelligence-ml/natural-language-processing-implementations/transformer-architectures.md) — Implements a transformer architecture utilizing multi-head attention and feed-forward networks for text generation.
- [Educational Implementations](https://awesome-repositories.com/f/artificial-intelligence-ml/large-language-models/educational-implementations.md) — Provides a from-scratch implementation of a generative transformer to teach neural network internals.
- [Educational Neural Network Implementations](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/frameworks/educational-neural-network-implementations.md) — Offers a pedagogical implementation of neural network components built from first principles.
- [Language Model Architectures](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/frameworks/model-construction/language-model-architectures.md) — Provides a framework for constructing language model architectures that predict subsequent tokens. ([source](https://github.com/karpathy/ng-video-lecture/search))
- [PyTorch Tensor Operations](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/model-optimization-and-inference/hardware-and-acceleration/tensor-computing-libraries/pytorch-tensor-operations.md) — Uses PyTorch for high-speed tensor operations and dynamic computational graph management.
- [Causal Masking](https://awesome-repositories.com/f/artificial-intelligence-ml/masked-language-modeling/causal-masking.md) — Implements causal masking to prevent the model from attending to future tokens during training.
- [Reference Models](https://awesome-repositories.com/f/artificial-intelligence-ml/pytorch-model-development/reference-models.md) — Implements a small-scale transformer model in PyTorch to demonstrate tokenization and self-attention.
- [Token Embedding Layers](https://awesome-repositories.com/f/artificial-intelligence-ml/vector-embeddings/dense-embeddings/token-embedding-layers.md) — Provides token embedding layers that map discrete characters to high-dimensional dense vectors.
- [Backpropagation](https://awesome-repositories.com/f/artificial-intelligence-ml/backpropagation.md) — Implements the backpropagation algorithm to update network weights by calculating loss function gradients.
- [Language Model Pre-training](https://awesome-repositories.com/f/artificial-intelligence-ml/language-model-pre-training.md) — Implements the pre-training process for a transformer model to learn general linguistic patterns. ([source](https://github.com/karpathy/ng-video-lecture/blob/master/README.md))
- [Multi-Layer Architectures](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/frameworks/model-construction/neural-network-layers/multi-layer-architectures.md) — Employs a multi-layer architecture by stacking identical transformer blocks to increase model depth.
- [Educational Model Training](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/model-training-and-tuning/training-frameworks/model-training-frameworks/vision-model-training/educational-model-training.md) — Executes a training process for a small-scale model to demonstrate machine learning convergence concepts.

### Part of an Awesome List

- [GPT Implementations](https://awesome-repositories.com/f/awesome-lists/ai/model-implementations/gpt-implementations.md) — Provides a concrete code implementation of the Generative Pre-trained Transformer architecture. ([source](https://github.com/karpathy/ng-video-lecture#readme))
- [Small Language Models](https://awesome-repositories.com/f/awesome-lists/ai/small-language-models.md) — Constructs a lightweight, small-scale language model for experimentation and learning.

### Education & Learning Resources

- [Generative AI Tutorials](https://awesome-repositories.com/f/education-learning-resources/curricula-instructional-design/curricula-roadmaps/ai-machine-learning-roadmaps/generative-ai-curricula/generative-ai-tutorials.md) — Provides a step-by-step tutorial on creating a text-generating neural network.
- [Educational Implementations](https://awesome-repositories.com/f/education-learning-resources/educational-resources/systems-applied-computing/machine-learning-education/llm-engineering-guides/transformer-model-tutorials/educational-implementations.md) — Serves as a functional, readable codebase for teaching the internal mechanics of transformer architectures.