5 repository-uri
Awesome GitHub RepositoriesTransformer Training Accelerators
GPU-optimized kernels and execution strategies specifically designed to improve throughput for transformer-based neural network training.
Distinguishing note: Distinct from general ML frameworks: targets specific hardware-level acceleration for transformer architectures.
Explore 5 awesome GitHub repositories matching artificial intelligence & ml · Transformer Training Accelerators. Refine with filters or upvote what's useful.
Awesome Transformer Training Accelerators GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- microsoft/deepspeed
microsoft/DeepSpeed
42,533Vezi pe GitHubDeepSpeed is a distributed deep learning optimization library and framework designed for the training and inference of massive AI models. It serves as a model parallelism orchestrator and a toolkit for scaling large language models across multiple GPUs and compute nodes. The project distinguishes itself through 3D parallelism orchestration, which combines data, pipeline, and tensor parallelism. It utilizes ZeRO-based memory partitioning to eliminate redundant storage and employs CPU-offload memory management to move weights and optimizer states to system RAM. Additionally, it provides special
Accelerates transformer training through specialized parallelism and dynamic sequence length optimization.
Python - deepspeedai/deepspeed
deepspeedai/DeepSpeed
42,528Vezi pe GitHubDeepSpeed is a high-performance library designed to scale deep learning model training and inference across massive clusters of GPUs and compute nodes. It provides a comprehensive suite of tools for distributed training, enabling the execution of models that exceed the memory capacity of single devices through advanced parameter partitioning, pipeline-based model parallelism, and memory-efficient state offloading. The framework distinguishes itself through specialized communication-efficient optimizers and hardware-aware acceleration techniques. By utilizing gradient compression, quantization
The framework accelerates transformer training by applying specialized GPU kernels that improve throughput on single devices and scale across multi-GPU clusters.
Pythonbillion-parameterscompressiondata-parallelism - dao-ailab/flash-attention
Dao-AILab/flash-attention
24,220Vezi pe GitHubFlashAttention is an attention mechanism optimization library and machine learning acceleration framework designed to increase training speed and reduce memory footprint for large-scale neural network models. It functions as a collection of low-level CUDA kernels that optimize memory-bound operations to improve hardware utilization on graphics processing units. The library distinguishes itself through an input-output-aware algorithm design that minimizes data movement between different levels of memory. By employing kernel fusion and tiled matrix multiplication, it combines sequential operati
Improves the performance of attention mechanisms to enable faster training and inference for transformer architectures.
Python - infrasys-ai/aisystem
Infrasys-AI/AISystem
17,017Vezi pe GitHubAISystem is a comprehensive AI full-stack infrastructure project covering the entire pipeline from AI chip architecture to high-level training frameworks. It encompasses the development of AI compiler frameworks, inference engines, and distributed training orchestrators designed to coordinate workloads across a heterogeneous compute stack of CPUs, GPUs, and NPUs. The project focuses on the deep integration of software and hardware, employing software-hardware co-design to align tensor layouts with physical memory structures. It provides specialized capabilities for accelerating Transformer mo
Provides a dedicated engine and optimized kernels to accelerate Transformer-based architectures and Mixture of Experts models.
Jupyter Notebookaiaiinfraaisys - linkedin/liger-kernel
linkedin/Liger-Kernel
6,148Vezi pe GitHubLiger-Kernel is a collection of pre-built fused Triton kernels and patching utilities designed to accelerate large language model training. It provides drop-in kernel replacements for common LLM operations such as RMSNorm, cross-entropy loss, and attention, enabling increased throughput and reduced memory usage while preserving bitwise-exact gradients. The project serves as a toolkit for composing custom model architectures from individual optimized kernels and for patching pre-existing models with minimal code changes. The project distinguishes itself through its ability to perform runtime m
Provides GPU-optimized Triton kernels that accelerate transformer training throughput by up to 20%.
Pythonfinetuninggemma2hacktoberfest
