How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
vLLM is a high-throughput inference engine designed for the efficient serving and execution of large language models. It functions as a production-ready distributed model server, providing standard API protocols for online serving while also supporting offline batch processing. The system is built to maximize token generation speed and memory efficiency, enabling both large-scale cloud deployments and local execution on personal hardware. The project distinguishes itself through advanced memory management and request scheduling techniques, most notably its use of non-contiguous key-value cach
llama.cpp is a high-performance C++ inference engine and runtime for executing large language models locally across various hardware architectures. It provides the core components for local model execution, including a dedicated model quantizer for compressing weights into the GGUF format and a system for generating text embeddings for semantic search. The project distinguishes itself through specialized memory and execution optimizations, such as block-wise weight quantization to reduce memory footprints and memory-mapped model loading. It supports structured text generation by using formal
OpenLLM is a framework for deploying, managing, and scaling open-source large language models
Flowise is a low-code platform designed for building and deploying complex language model workflows through a visual, node-based interface. It functions as an orchestrator for autonomous multi-agent systems, allowing users to construct conversational pipelines by connecting language models, memory stores, and external tools on a drag-and-drop canvas. The platform distinguishes itself through its support for sophisticated agentic patterns, including supervisor-worker delegation and iterative reasoning strategies. Users can design directed acyclic graphs to manage conditional branching, state p
MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
The main features of microsoft/deepspeed-mii are: Inference and Serving, Model Serving Engines, Inference Frameworks.
Projects with overlapping indexed features include: vllm-project/vllm — vLLM is a high-throughput inference engine designed for the efficient serving and execution of large language models.… huggingface/text-generation-inference — Text Generation Inference is a production-ready engine designed for the deployment and serving of large language… flowiseai/flowise — Flowise is a low-code platform designed for building and deploying complex language model workflows through a visual,… bentoml/openllm — OpenLLM is a framework for deploying, managing, and scaling open-source large language models. huggingface/text-embeddings-inference — Text Embeddings Inference is a high-performance inference server designed to host text embedding and sequence… ggerganov/llama.cpp — llama.cpp is a high-performance C++ inference engine and runtime for executing large language models locally across…