How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
A throughput-oriented high-performance serving framework for LLMs
FlashInfer is a library of high-performance GPU kernels purpose-built for accelerating large language model inference. It provides optimized implementations for attention operations (including flash attention, page attention, multi-head latent attention, and cascade attention) using paged key-value caches, fused kernel composition, and just-in-time compilation. The library also includes specialized kernels for mixture-of-experts layers, block-scaled low-precision quantization (FP8, FP4), and distributed collective communication. What distinguishes FlashInfer is its fused all-reduce communicat
Automatically Discovering Fast Parallelization Strategies for Distributed Deep Neural Network Training
Petals is a decentralized framework and inference engine for running large language models across a peer-to-peer network. It enables the execution of models that exceed the memory of any single machine by splitting computations and model layers across a collaborative swarm of GPUs. The system functions as a collaborative compute network where participants share local GPU resources and host model weights. It supports distributed prompt-tuning to adapt massive models to specific tasks and allows for the establishment of private compute swarms to process sensitive data within restricted, trusted
Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).
The main features of inferflow/inferflow are: Inference Frameworks.
Open-source alternatives to inferflow/inferflow include: efeslab/nanoflow — A throughput-oriented high-performance serving framework for LLMs. flashinfer-ai/flashinfer — FlashInfer is a library of high-performance GPU kernels purpose-built for accelerating large language model inference.… flexflow/flexflow — Automatically Discovering Fast Parallelization Strategies for Distributed Deep Neural Network Training. fminference/flexgen — FlexGen is an inference engine for large language models designed for high-throughput execution on single or multiple… ggerganov/llama.cpp — llama.cpp is a high-performance C++ inference engine and runtime for executing large language models locally across… bigscience-workshop/petals — Petals is a decentralized framework and inference engine for running large language models across a peer-to-peer…