Open-source alternatives to Flashinfer

LeetCUDA is a collection of high-performance GPU kernel libraries focusing on memory optimization, activation functions, and attention mechanisms. It serves as a reference library for CUDA kernel implementations, ranging from basic element-wise operations to complex neural network components, and provides Python bindings to integrate these kernels into deep learning workflows. The project is distinguished by its focus on low-level hardware optimizations. This includes the use of tensor cores for half-precision matrix multiplication, asynchronous data pipelining with double buffering, and shar

This project provides a comprehensive technical guide and framework for engineering large-scale machine learning systems. It covers the full lifecycle of model development, focusing on the infrastructure and computational principles required to build, train, and serve generative AI models across distributed GPU clusters. The repository distinguishes itself by offering deep-dive tutorials and implementation strategies for complex system challenges. It emphasizes high-performance architectural primitives, such as collective communication orchestration, distributed tensor sharding, and static gr

Transformers is a comprehensive library for machine learning that provides a unified interface for training, fine-tuning, and deploying transformer-based models. It supports a wide range of tasks, including text classification, language modeling, question answering, and sequence-to-sequence translation, while offering specialized architectures for both text and vision processing. The framework includes tools for managing the entire model lifecycle, from data preprocessing and tokenization to distributed training and inference. The library features extensive support for model optimization and

jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti

xformers is a collection of specialized toolsets for fused GPU operators, sparse attention mechanisms, modular transformer components, and performance benchmarking. It provides a library of optimized and interoperable building blocks used to construct and experiment with transformer architectures. The project features a fused CUDA operator library that combines common layers into single GPU operations to increase throughput. It includes a sparse attention framework and memory-efficient attention kernels that utilize tiling strategies and structured sparsity patterns to reduce computational ov

DeepSeek-V2 is a large language model designed for natural language processing and the analysis of long text sequences. It utilizes a mixture-of-experts architecture to balance high performance with inference efficiency. The model employs a sparse routing mechanism and shared expert neurons to capture common knowledge while maintaining specialization. It further reduces memory overhead and increases throughput through multi-head latent attention, group-query attention, and low-rank tensor compression. These capabilities enable the processing and retrieval of information from extensive token

TileLang is a Python-embedded domain-specific language compiler that JIT-compiles and autotunes GPU kernels. It uses a tile-based DSL, automatic software pipelining, and parallel autotuning to generate optimized GPU kernels at runtime. It supports tensor core operations with Pythonic syntax, automatic memory management, and thread mapping. The compiler searches over tile sizes, thread counts, and scheduling policies, compiling and benchmarking candidates in parallel to find the fastest kernel. It also caches compiled binaries and tuning results to disk for reuse across sessions. TileLang inc

Flux.jl is a deep learning framework and numerical computing toolkit written in Julia. It serves as a machine learning library for designing and training neural networks, providing a system for automatic differentiation to optimize model parameters. The framework enables deep learning development and machine learning research by representing layers as parameterized functions. It supports scientific machine learning, integrating neural networks into workflows for solving physical and mathematical problems. The toolkit provides native GPU acceleration for tensor computations and utilizes rever

This project is a CUDA programming course and technical guide focused on writing and optimizing GPU kernels for hardware acceleration. It provides structured learning resources for using the CUDA platform to execute operations on silicon architectures. The material covers the optimization of linear algebra kernels and the analysis of machine learning deployment. It includes guidance on identifying acceleration tools, mapping the deep learning ecosystem, and evaluating the frameworks used to move models from research to production environments. The scope extends to GPU performance optimizatio

This project is a high-performance C++ and CUDA neural network library designed for fast training and inference of small networks on NVIDIA GPUs. It serves as a specialized backend for neural radiance fields and coordinate-based networks, providing a fused GPU kernel library and a hash grid encoder for transforming raw input dimensions into high-dimensional representations. The library distinguishes itself through the use of C++ template metaprogramming and fused-kernel execution, which merge neural network layers into single GPU device functions to eliminate memory bottlenecks. It leverages

MergeKit is a toolkit for combining multiple pre-trained large language models into a single entity using algorithmic blending. It provides a specialized system for parameter interpolation and weight extraction to unify model capabilities. The project distinguishes itself through an evolutionary merge optimizer that tunes parameters based on quantitative evaluation metrics. It also features a mixture of experts orchestrator capable of converting dense models into sparse architectures and a tokenizer alignment tool for transplanting embeddings between different models. The toolkit covers a br

rust-cuda is a GPU programming framework and device compiler that allows for the development and execution of high-performance kernels on NVIDIA hardware using Rust. It provides a driver wrapper to manage device memory allocation and kernel launching, effectively serving as a system for writing GPU compute logic without relying on C++. The project includes a compute library with hardware-optimized primitives for neural network acceleration and hardware-accelerated raytracing. It utilizes a compilation toolchain that translates source code into a low-level intermediate representation for execu

Grok-1 is an open-weights large language model implementation featuring a sparse mixture-of-experts architecture. It is designed for high-performance text generation and natural language processing by activating only a subset of specialized expert layers per token. The model utilizes 8-bit weight quantization to reduce memory overhead and accelerate loading. To manage its high parameter count, the implementation supports activation sharding, which distributes the memory load across multiple hardware devices during execution. The project covers large-scale model inference, including text comp

Neon is a deep learning framework and hardware-abstraction machine learning stack used for designing, training, and deploying neural network architectures. It functions as a graph-based computation engine that utilizes just-in-time kernel compilation to optimize machine code for tensors. The platform decouples model definitions from execution kernels, allowing it to support multiple CPU and GPU backends. This architecture enables the distribution of computational workloads across parallelized hardware environments to increase processing speed and overall efficiency. The system covers the ful

Segment Anything Fast is a high-performance computer vision inference engine and image segmentation framework built for PyTorch. It provides a specialized environment for automated object isolation and mask generation, designed to process large-scale visual datasets with increased throughput. The project distinguishes itself through a suite of system-level optimization strategies that accelerate deep learning model performance. By utilizing graph-based model compilation, just-in-time kernel fusion, and hardware-aware quantization, it reduces computational latency and memory footprint. These t

FlashMLA is an LLM attention kernel library and inference acceleration library providing a collection of high-performance CUDA kernels. It implements multi-head latent attention mechanisms designed to reduce memory overhead and increase throughput during the forward and backward passes of large language model inference. The library utilizes quantized cache attention kernels to improve computation efficiency across both sparse and dense token processing. It specifically optimizes the prefill and decoding phases of model inference through these latent attention implementations. The project cov

AITemplate is an ahead-of-time deep learning compiler that translates PyTorch neural networks into standalone C++ source code. It functions as a PyTorch to C++ compiler and a GPU kernel fusion engine, producing self-contained executable binaries that run inference without requiring a Python interpreter or deep learning framework runtime. The project generates optimized CUDA and HIP C++ code specifically for NVIDIA TensorCores and AMD MatrixCores. It focuses on maximizing throughput for half-precision floating-point operations through a system that combines multiple neural network operators in

This project is a comprehensive technical course study guide and reference for learning the architectures and training methods of Transformers and large language models. It serves as a technical overview for understanding how neural networks process data and how to align model behavior with specific performance goals. The repository provides specialized guides on several key areas of model development. This includes detailed references for transformer architectures, implementation frameworks for retrieval-augmented generation and agentic workflows, and technical guides for model optimization

DeepGEMM is a suite of specialized GPU kernels and a just-in-time compiler designed for low-precision matrix operations, Mixture-of-Experts models, and attention processing. It provides a library of high-performance matrix multiplication kernels using FP8 precision to increase compute throughput and reduce memory usage. The project features a JIT CUDA kernel compiler that generates and loads optimized compute kernels at runtime to eliminate the need for manual compilation during installation. It includes specialized implementations for grouped matrix multiplication that process multiple group

Flash Linear Attention is a training framework and inference engine for sequence models that use linear attention and state space mechanisms, designed to process long contexts with reduced memory and compute overhead. It provides hardware-optimized token mixing layers and fused CUDA kernels that minimize memory bandwidth and launch overhead across different GPU architectures, and includes a causal inference engine that generates text token-by-token using cached hidden states for efficient autoregressive decoding. The project supports building hybrid sequence models that interleave standard at

Mergekit is a toolkit for combining multiple pretrained large language models into a single model. It functions as an architecture assembler and merging system that transfers capabilities between models using weighted algorithms and layer-wise assembly without requiring additional training. The project provides specialized utilities for extracting low-rank approximations from fine-tuned models to create portable parameter updates. It also includes a framework for converting dense language models into a mixture of experts architecture by constructing gating mechanisms to route inputs to specia

OpenMythos is a framework for implementing recurrent large language model architectures. It utilizes recurrent transformer blocks to enable compute-adaptive reasoning and variable processing depth through multiple iterative passes over the same weights. The system features a mixture of experts framework that routes tokens between shared and specialized layers to optimize parameter usage. It also includes parameter-efficient fine-tuning tools using low-rank adaptation modules to modify model behavior with minimal weight updates. The framework covers distributed training pipelines using data p

AutoGPTQ is a model compression toolkit and post-training quantization framework designed to reduce the memory footprint of large language models. It utilizes the GPTQ algorithm to compress neural network weights, lowering hardware requirements and reducing VRAM usage. The project serves as an inference accelerator by providing optimized kernels that increase token generation speed. It features model architecture extensibility, allowing quantization capabilities to be added to new model structures through configurable patterns. The framework covers a comprehensive quantization pipeline, incl

JAX is a hardware-accelerated array library and automatic differentiation system for numerical computing. It provides a framework compatible with NumPy that extends array operations with a just-in-time compiler to transform Python functions into optimized kernels for execution on GPU and TPU accelerators. The system differentiates itself through the use of an XLA-based compiler and a single program multiple data sharding model. These capabilities allow the library to distribute large-scale computations across multiple hardware accelerators using both automatic parallelization and manual shard

DeepEP is a distributed model accelerator and expert-parallel communication library designed to optimize the training and inference of large-scale neural networks. It provides specialized GPU communication kernels and a remote GPU memory interface to facilitate high-throughput data exchange between hardware nodes. The system utilizes dynamic kernel generation to compile optimized GPU kernels during execution, removing the need for separate installation compilation steps. It implements virtual-lane traffic isolation to prevent interference between different data streams and employs routing met

Apex is a high-performance toolkit for PyTorch designed to coordinate distributed training, execute fused GPU kernels, manage mixed precision, and implement optimized distributed optimizers. It provides specialized tools for scaling model training across multiple GPUs and nodes to increase processing speed and throughput. The library features high-performance implementations of Adam and LAMB optimizers to reduce synchronization overhead and memory bottlenecks. It utilizes fused CUDA kernels to combine neural network operations, reducing memory overhead and increasing execution speed. The too