How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
PowerInfer is a high-performance local large language model inference engine and sparse inference framework. It provides a runtime for executing models on consumer-grade hardware, utilizing a GPU acceleration backend to optimize tensor operations for graphics processors. The system distinguishes itself through a sparse inference framework that increases generation speed by skipping computations based on activation sparsity in model weights. It includes a GGUF model converter for transforming weights and metadata into a unified binary format, as well as an OpenAI API compatible server for inte
PowerInfer is an inference engine and serving framework designed to run large language models on local hardware. It combines a hybrid CPU-GPU offloader, a quantization tool, and a sparse model optimizer to enable the execution of high-parameter models on consumer-grade devices. The system distinguishes itself through neuron-activation-based offloading, using a predictor model to preload frequent neurons into VRAM while keeping rare neurons in system memory. This hybrid execution model balances workloads between the GPU and CPU based on input patterns to optimize memory access and increase tok
DeepSpeed is a distributed deep learning optimization library and framework designed for the training and inference of massive AI models. It serves as a model parallelism orchestrator and a toolkit for scaling large language models across multiple GPUs and compute nodes. The project distinguishes itself through 3D parallelism orchestration, which combines data, pipeline, and tensor parallelism. It utilizes ZeRO-based memory partitioning to eliminate redundant storage and employs CPU-offload memory management to move weights and optimizer states to system RAM. Additionally, it provides special
FlexLLMGen is an inference engine and runtime designed to run large language models on a single GPU by combining weight compression with tensor offloading. It reduces model weight memory usage by approximately 70% through 4-bit quantization, and stores model parameters, attention cache, and hidden states across GPU, CPU, and disk to fit models larger than available GPU memory. The project distinguishes itself through a throughput-oriented batching approach that processes multiple generation requests together in large batches to maximize throughput on a single GPU. It also supports distributed
FlexGen is an inference engine for large language models designed for high-throughput execution on single or multiple GPUs. It functions as a framework for managing model execution through a combination of memory offloading, weight compression, and pipeline orchestration.
The main features of fminference/flexgen are: Efficient Inference Engines, High Throughput Inference, Large Language Model Optimization, Parallelism Orchestration, Weight Offloading, Weight Quantization, Quantized Model Runners, Model Layer Offloading.
Open-source alternatives to fminference/flexgen include: tiiny-ai/powerinfer — PowerInfer is a high-performance local large language model inference engine and sparse inference framework. It… sjtu-ipads/powerinfer — PowerInfer is an inference engine and serving framework designed to run large language models on local hardware. It… microsoft/deepspeed — DeepSpeed is a distributed deep learning optimization library and framework designed for the training and inference of… fminference/flexllmgen — FlexLLMGen is an inference engine and runtime designed to run large language models on a single GPU by combining… openbmb/minicpm — MiniCPM is a collection of small language models designed for local, on-device deployment in resource-constrained… vllm-project/llm-compressor — llm-compressor is a quantization toolkit and post-training library designed to reduce the memory footprint and size of…