A throughput-oriented high-performance serving framework for LLMs
FlashInfer is a library of high-performance GPU kernels purpose-built for accelerating large language model inference. It provides optimized implementations for attention operations (including flash attention, page attention, multi-head latent attention, and cascade attention) using paged key-value caches, fused kernel composition, and just-in-time compilation. The library also includes specialized kernels for mixture-of-experts layers, block-scaled low-precision quantization (FP8, FP4), and distributed collective communication. What distinguishes FlashInfer is its fused all-reduce communicat
Automatically Discovering Fast Parallelization Strategies for Distributed Deep Neural Network Training
Petals is a decentralized framework and inference engine for running large language models across a peer-to-peer network. It enables the execution of models that exceed the memory of any single machine by splitting computations and model layers across a collaborative swarm of GPUs. The system functions as a collaborative compute network where participants share local GPU resources and host model weights. It supports distributed prompt-tuning to adapt massive models to specific tasks and allows for the establishment of private compute swarms to process sensitive data within restricted, trusted
Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).
inferflow/inferflow 的主要功能包括：Inference Frameworks。
inferflow/inferflow 的开源替代品包括： efeslab/nanoflow — A throughput-oriented high-performance serving framework for LLMs. flashinfer-ai/flashinfer — FlashInfer is a library of high-performance GPU kernels purpose-built for accelerating large language model inference.… flexflow/flexflow — Automatically Discovering Fast Parallelization Strategies for Distributed Deep Neural Network Training. fminference/flexgen — FlexGen is an inference engine for large language models designed for high-throughput execution on single or multiple… ggerganov/llama.cpp — llama.cpp is a high-performance C++ inference engine and runtime for executing large language models locally across… bigscience-workshop/petals — Petals is a decentralized framework and inference engine for running large language models across a peer-to-peer…