3 repository-uri
Awesome GitHub RepositoriesGradient Checkpointing
Techniques that reduce peak memory consumption by recomputing intermediate activations during computational processes.
Explore 3 awesome GitHub repositories matching operating systems & systems programming · Gradient Checkpointing. Refine with filters or upvote what's useful.
Awesome Gradient Checkpointing GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- unslothai/unsloth
unslothai/unsloth
66,628Vezi pe GitHubUnsloth is a high-performance training and inference platform designed to optimize the lifecycle of large language and multimodal models. It provides a comprehensive engine for fine-tuning, executing, and managing models locally, with a focus on reducing memory consumption and increasing compute speed on consumer-grade hardware. The platform distinguishes itself through hand-optimized kernels and automated computational graph techniques that maximize hardware throughput. It supports advanced training methodologies, including reinforcement learning for reasoning and efficient adapter-based fin
Lowers peak memory consumption by recomputing intermediate activations during the backward pass instead of storing them.
Pythonagentdeepseekdeepseek-r1 - karpathy/llm.c
karpathy/llm.c
30,230Vezi pe GitHubThis project is a low-dependency engine designed for training large language models using native C and CUDA. It provides a bare-metal environment for tensor computation, allowing for the execution of neural network operations directly on hardware accelerators without the overhead of high-level software abstractions. The framework distinguishes itself by implementing manual gradient backpropagation and custom hardware-specific kernels, providing granular control over memory mapping and computational precision. It supports distributed training across multiple graphics processors and compute nod
Utilizes gradient checkpointing to reduce memory consumption by recomputing intermediate activations during the backward pass.
Cuda - dao-ailab/flash-attention
Dao-AILab/flash-attention
24,220Vezi pe GitHubFlashAttention is an attention mechanism optimization library and machine learning acceleration framework designed to increase training speed and reduce memory footprint for large-scale neural network models. It functions as a collection of low-level CUDA kernels that optimize memory-bound operations to improve hardware utilization on graphics processing units. The library distinguishes itself through an input-output-aware algorithm design that minimizes data movement between different levels of memory. By employing kernel fusion and tiled matrix multiplication, it combines sequential operati
Reduces memory usage by discarding intermediate attention scores and recomputing them during the backward pass.
Python
