TensorRT-LLM is a platform and toolkit designed for compiling, optimizing, and serving transformer-based models on accelerated hardware. It functions as a framework that transforms machine learning models into efficient execution graphs, providing an engine to refine these models for specific hardware to maximize throughput and minimize latency during text generation. The project distinguishes itself through advanced execution strategies that manage the entire inference pipeline. It utilizes kernel-level fusion and static graph execution to optimize mathematical operations and computational f
FlashInfer is a library of high-performance GPU kernels purpose-built for accelerating large language model inference. It provides optimized implementations for attention operations (including flash attention, page attention, multi-head latent attention, and cascade attention) using paged key-value caches, fused kernel composition, and just-in-time compilation. The library also includes specialized kernels for mixture-of-experts layers, block-scaled low-precision quantization (FP8, FP4), and distributed collective communication. What distinguishes FlashInfer is its fused all-reduce communicat
Automatically Discovering Fast Parallelization Strategies for Distributed Deep Neural Network Training
A throughput-oriented high-performance serving framework for LLMs
Official repository for DistFlashAttn: Distributed Memory-efficient Attention for Long-context LLMs Training
rulinshao/lightseq 的主要功能包括：Inference Frameworks, Inference Serving Engines。
rulinshao/lightseq 的开源替代品包括： nvidia/tensorrt-llm — TensorRT-LLM is a platform and toolkit designed for compiling, optimizing, and serving transformer-based models on… flashinfer-ai/flashinfer — FlashInfer is a library of high-performance GPU kernels purpose-built for accelerating large language model inference.… flexflow/flexflow — Automatically Discovering Fast Parallelization Strategies for Distributed Deep Neural Network Training. fminference/flexgen — FlexGen is an inference engine for large language models designed for high-throughput execution on single or multiple… ggerganov/llama.cpp — llama.cpp is a high-performance C++ inference engine and runtime for executing large language models locally across… efeslab/nanoflow — A throughput-oriented high-performance serving framework for LLMs.