# karpathy/nn-zero-to-hero

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/karpathy-nn-zero-to-hero).**

20,351 stars · 2,898 forks · Jupyter Notebook · mit

## Links

- GitHub: https://github.com/karpathy/nn-zero-to-hero
- awesome-repositories: https://awesome-repositories.com/repository/karpathy-nn-zero-to-hero.md

## Description

This project is an educational resource and pedagogical framework designed to teach the fundamental mechanics of neural networks and gradient-based optimization. It provides a series of tutorials and code examples that guide users through building deep learning models from scratch, focusing on the implementation of core mathematical primitives and the underlying logic of backpropagation.

The project distinguishes itself by providing a custom automatic differentiation engine that tracks mathematical operations in a dynamic computational graph. By implementing reverse-mode automatic differentiation and topological sort execution, it allows users to compute gradients for complex expressions without manual derivation, providing a transparent view into how neural network architectures are structured and trained.

The repository covers the foundational aspects of machine learning, including the construction of layers and activation functions using scalar-based primitive operations. These tools enable the manual assembly of neural networks, facilitating a conceptual understanding of how systems learn patterns and perform predictions. The content is delivered through a series of Jupyter Notebooks that serve as a structured course on deep learning mechanics.

## Tags

### Artificial Intelligence & ML

- [Automatic Differentiation Engines](https://awesome-repositories.com/f/artificial-intelligence-ml/automatic-differentiation-engines.md) — Ships a pedagogical automatic differentiation engine for training neural networks.
- [Gradient Computation](https://awesome-repositories.com/f/artificial-intelligence-ml/gradient-computation.md) — Calculates gradients automatically to facilitate efficient model optimization. ([source](https://github.com/karpathy/nn-zero-to-hero/tree/master/lectures/))
- [Neural Network Implementations](https://awesome-repositories.com/f/artificial-intelligence-ml/neural-network-implementations.md) — Guides the manual implementation of neural network architectures, layers, and activation functions.
- [Reverse-Mode Differentiation](https://awesome-repositories.com/f/artificial-intelligence-ml/reverse-mode-differentiation.md) — Implements reverse-mode automatic differentiation to propagate error signals and update model parameters.
- [Neural Network Layers](https://awesome-repositories.com/f/artificial-intelligence-ml/neural-network-layers.md) — Enables the construction of custom neural network layers and architectures using basic primitives. ([source](https://github.com/karpathy/nn-zero-to-hero/tree/master/lectures/))
- [Dynamic Graph Builders](https://awesome-repositories.com/f/artificial-intelligence-ml/gradient-computation/dynamic-graph-builders.md) — Constructs computational graphs dynamically at runtime to support flexible model architectures.
- [Computational Graphs](https://awesome-repositories.com/f/artificial-intelligence-ml/computational-graphs.md) — Provides a mechanism for traversing computational graphs to calculate gradients during the backpropagation process.

### Education & Learning Resources

- [Deep Learning Courses](https://awesome-repositories.com/f/education-learning-resources/deep-learning-courses.md) — Delivers a structured course on deep learning mechanics through tutorials and code examples.
- [Deep Learning Platforms](https://awesome-repositories.com/f/education-learning-resources/deep-learning-education/deep-learning-platforms.md) — Serves as a structured educational platform for building neural networks from scratch.
- [Neural Network Implementations](https://awesome-repositories.com/f/education-learning-resources/educational-resources/reference-and-media/books-docs-reference/code-examples/reference-implementations/neural-network-implementations.md) — Provides educational code examples for building and optimizing neural networks from scratch.
- [Machine Learning Fundamentals](https://awesome-repositories.com/f/education-learning-resources/technical-domain-education/ai-machine-learning-education/machine-learning-fundamentals.md) — Teaches machine learning fundamentals through hands-on implementation of core algorithms.