How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
A throughput-oriented high-performance serving framework for LLMs
FlashInfer is a library of high-performance GPU kernels purpose-built for accelerating large language model inference. It provides optimized implementations for attention operations (including flash attention, page attention, multi-head latent attention, and cascade attention) using paged key-value caches, fused kernel composition, and just-in-time compilation. The library also includes specialized kernels for mixture-of-experts layers, block-scaled low-precision quantization (FP8, FP4), and distributed collective communication. What distinguishes FlashInfer is its fused all-reduce communicat
Automatically Discovering Fast Parallelization Strategies for Distributed Deep Neural Network Training
Petals is a decentralized framework and inference engine for running large language models across a peer-to-peer network. It enables the execution of models that exceed the memory of any single machine by splitting computations and model layers across a collaborative swarm of GPUs. The system functions as a collaborative compute network where participants share local GPU resources and host model weights. It supports distributed prompt-tuning to adapt massive models to specific tasks and allows for the establishment of private compute swarms to process sensitive data within restricted, trusted
计图大模型推理库，具有高性能、配置要求低、中文支持好、可移植等特点
The main features of jittor/jittorllms are: Inference Frameworks, LLM Utilities.
Projects with overlapping indexed features include: efeslab/nanoflow — A throughput-oriented high-performance serving framework for LLMs. flashinfer-ai/flashinfer — FlashInfer is a library of high-performance GPU kernels purpose-built for accelerating large language model inference.… flexflow/flexflow — Automatically Discovering Fast Parallelization Strategies for Distributed Deep Neural Network Training. fminference/flexgen — FlexGen is an inference engine for large language models designed for high-throughput execution on single or multiple… ggerganov/llama.cpp — llama.cpp is a high-performance C++ inference engine and runtime for executing large language models locally across… bigscience-workshop/petals — Petals is a decentralized framework and inference engine for running large language models across a peer-to-peer…