karpathy/nn-zero-to-hero
Nn Zero To Hero
This project is an educational resource and pedagogical framework designed to teach the fundamental mechanics of neural networks and gradient-based optimization. It provides a series of tutorials and code examples that guide users through building deep learning models from scratch, focusing on the implementation of core mathematical primitives and the underlying logic of backpropagation.
The project distinguishes itself by providing a custom automatic differentiation engine that tracks mathematical operations in a dynamic computational graph. By implementing reverse-mode automatic differentiation and topological sort execution, it allows users to compute gradients for complex expressions without manual derivation, providing a transparent view into how neural network architectures are structured and trained.
The repository covers the foundational aspects of machine learning, including the construction of layers and activation functions using scalar-based primitive operations. These tools enable the manual assembly of neural networks, facilitating a conceptual understanding of how systems learn patterns and perform predictions. The content is delivered through a series of Jupyter Notebooks that serve as a structured course on deep learning mechanics.
Features
- Automatic Differentiation Engines - Ships a pedagogical automatic differentiation engine for training neural networks.
- Deep Learning Courses - Delivers a structured course on deep learning mechanics through tutorials and code examples.
- Deep Learning Platforms - Serves as a structured educational platform for building neural networks from scratch.
- Neural Network Implementations - Provides educational code examples for building and optimizing neural networks from scratch.
- Gradient Computation - Calculates gradients automatically to facilitate efficient model optimization.
- Neural Network Implementations - Guides the manual implementation of neural network architectures, layers, and activation functions.
- Reverse-Mode Differentiation - Implements reverse-mode automatic differentiation to propagate error signals and update model parameters.
- Neural Network Layers - Enables the construction of custom neural network layers and architectures using basic primitives.
- Deep Learning Foundations - Educational implementation of neural networks from first principles.
- Machine Learning Fundamentals - Teaches machine learning fundamentals through hands-on implementation of core algorithms.
- Dynamic Graph Builders - Constructs computational graphs dynamically at runtime to support flexible model architectures.
- Computational Graphs - Provides a mechanism for traversing computational graphs to calculate gradients during the backpropagation process.
