# rasbt/llms-from-scratch

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/rasbt-llms-from-scratch).**

97,260 stars · 14,878 forks · Jupyter Notebook · NOASSERTION

## Links

- GitHub: https://github.com/rasbt/LLMs-from-scratch
- Homepage: https://amzn.to/4fqvn0D
- awesome-repositories: https://awesome-repositories.com/repository/rasbt-llms-from-scratch.md

## Topics

`ai` `artificial-intelligence` `chatbot` `chatgpt` `deep-learning` `from-scratch` `generative-ai` `gpt` `language-model` `large-language-models` `llm` `machine-learning` `neural-networks` `python` `pytorch` `transformers`

## Description

This repository serves as an educational framework for building large language models from the ground up. It provides a structured curriculum that guides learners through the end-to-end lifecycle of model development, including data processing, architecture design, and optimization. By focusing on low-level implementation, the project enables users to master the fundamental mechanics of artificial intelligence without relying on high-level abstraction frameworks.

The project distinguishes itself by constructing neural network components and gradient-based optimization logic from first principles. It utilizes tensor-based computational modeling and stateless functional architectures to define network layers as pure mathematical transformations. This approach exposes the underlying mechanics of weight updates and loss minimization, allowing for a deeper conceptual mastery of modern machine learning architectures.

The content is organized into a series of executable notebooks that facilitate incremental learning. Each chapter is encapsulated within an independent directory, providing a clear separation of concerns that simplifies dependency management. The repository supports various execution environments, including local Python, Docker containers, and cloud-based platforms, ensuring that the code remains accessible and functional on conventional hardware.

## Tags

### Artificial Intelligence & ML

- [Generative AI Resources](https://awesome-repositories.com/f/artificial-intelligence-ml/generative-ai-resources.md) — Guides learners through the end-to-end creation of generative language models using a structured, ground-up approach.
- [Backpropagation Implementations](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/architectures/neural-network-components/backpropagation-implementations.md) — Implements gradient-based optimization logic manually to clarify the mechanics of weight updates and loss minimization.
- [Deep Learning Implementations](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/architectures/neural-network-components/deep-learning-implementations.md) — Translates complex deep learning theory into functional code to provide practical experience with neural network architectures.
- [Educational Neural Network Implementations](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/frameworks/educational-neural-network-implementations.md) — Demonstrates the construction of neural network components from first principles without relying on high-level abstractions.
- [Model Training Frameworks](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/model-training-and-tuning/training-frameworks/model-training-frameworks.md) — Establishes a structured environment for building and training custom language models to master the development lifecycle.
- [Neural Network Components](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/architectures/neural-network-components.md) — Defines modular mathematical transformations and layer structures essential for building custom neural network architectures.

### Education & Learning Resources

- [Large Language Model Tutorials](https://awesome-repositories.com/f/education-learning-resources/educational-resources/reference-and-media/tutorials-media-curated-lists/technical-tutorials/machine-learning-ai/large-language-model-tutorials.md) — Explains the fundamental mechanics of large language models through hands-on, step-by-step implementation examples.
- [Machine Learning Curricula](https://awesome-repositories.com/f/education-learning-resources/machine-learning-curricula.md) — Offers a comprehensive educational path covering the architecture, training, and implementation of large language models.
- [Technical Training Repositories](https://awesome-repositories.com/f/education-learning-resources/educational-resources/courses-training-certifications/courses-structured-learning/courses/technical-training-repositories.md) — Bundles instructional materials and code examples to guide developers through the implementation of complex technical systems.
- [Interactive Notebooks](https://awesome-repositories.com/f/education-learning-resources/interactive-notebooks.md) — Organizes technical concepts into sequential, executable notebooks that allow users to verify theory through immediate practice.
- [Interactive Learning Platforms](https://awesome-repositories.com/f/education-learning-resources/educational-resources/courses-training-certifications/interactive-learning-platforms.md) — Provides interactive, executable documents that allow users to experiment with model components in a live environment.
- [Technical Tutorials](https://awesome-repositories.com/f/education-learning-resources/educational-resources/reference-and-media/tutorials-media-curated-lists/technical-tutorials.md) — Teaches the fundamentals of large language models through a structured curriculum of guided, hands-on tutorials. ([source](https://cdn.jsdelivr.net/gh/rasbt/LLMs-from-scratch@main/README.md))
- [Video Courses](https://awesome-repositories.com/f/education-learning-resources/educational-resources/reference-and-media/tutorials-media-curated-lists/interactive-learning-media/video-courses.md) — Features a comprehensive companion video course that walks through the implementation of each chapter. ([source](https://cdn.jsdelivr.net/gh/rasbt/LLMs-from-scratch@main/README.md))

### Scientific & Mathematical Computing

- [Low-Level Tensor Libraries](https://awesome-repositories.com/f/scientific-mathematical-computing/high-performance-execution-environments/scientific-computing-platforms/low-level-tensor-libraries.md) — Utilizes low-level array manipulation to perform mathematical operations and build neural network layers from scratch.

### Part of an Awesome List

- [AI and Neural Networks](https://awesome-repositories.com/f/awesome-lists/ai/ai-and-neural-networks.md) — Building large language models from the ground up.
- [Artificial Intelligence](https://awesome-repositories.com/f/awesome-lists/ai/artificial-intelligence.md) — Listed in the “Artificial Intelligence” section of the Build Your Own X awesome list.
- [LLM Development and Research](https://awesome-repositories.com/f/awesome-lists/ai/llm-development-and-research.md) — Step-by-step implementation of a ChatGPT-like model.
- [Machine Learning](https://awesome-repositories.com/f/awesome-lists/ai/machine-learning.md) — Educational repository for building LLMs.
- [Educational Resources](https://awesome-repositories.com/f/awesome-lists/learning/educational-resources.md) — Step-by-step guide to building a language model from the ground up.

### Software Engineering & Architecture

- [Modular and Plugin Architectures](https://awesome-repositories.com/f/software-engineering-architecture/software-architecture/architectural-patterns/plugin-module-systems/modular-plugin-architectures.md) — Separates distinct stages of model development into independent, modular directories for clear architectural organization.