How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
OpenLLM is a framework for deploying, managing, and scaling open-source large language models
The main features of bentoml/openllm are: Serving Frameworks, Large Language Models, Model Inference Servers, Model Gateways, AI Infrastructure Managers, AI Workflow Orchestrators, LLM Application Orchestration, Model Serving APIs.
Projects with overlapping indexed features include: sgl-project/sglang — Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It… vllm-project/vllm — vLLM is a high-throughput inference engine designed for the efficient serving and execution of large language models.… huggingface/text-generation-inference — Text Generation Inference is a production-ready engine designed for the deployment and serving of large language… kvcache-ai/ktransformers — Ktransformers is a comprehensive framework designed for the operation, fine-tuning, and serving of large language… berriai/litellm — LiteLLM is a unified gateway and proxy server designed to centralize access to over one hundred language model… open-webui/open-webui — Open WebUI is a self-hosted, web-based platform designed for interacting with local and remote artificial intelligence…
Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr
vLLM is a high-throughput inference engine designed for the efficient serving and execution of large language models. It functions as a production-ready distributed model server, providing standard API protocols for online serving while also supporting offline batch processing. The system is built to maximize token generation speed and memory efficiency, enabling both large-scale cloud deployments and local execution on personal hardware. The project distinguishes itself through advanced memory management and request scheduling techniques, most notably its use of non-contiguous key-value cach
Text Generation Inference is a production-ready engine designed for the deployment and serving of large language models. It functions as a containerized runtime environment that manages model execution, scales across distributed hardware, and provides high-performance inference capabilities for demanding production environments. The project distinguishes itself through advanced optimization techniques, including continuous batching to maximize hardware utilization and tensor parallelism to shard large models across multiple accelerator cards. It supports efficient inference through custom com
Ktransformers is a comprehensive framework designed for the operation, fine-tuning, and serving of large language models. It functions as a heterogeneous inference engine and quantized execution runtime, enabling the deployment of massive models by distributing computational workloads across both CPU and GPU resources. This architecture allows users to bypass local memory constraints, making it possible to run and train models that exceed the capacity of a single device. The project distinguishes itself through specialized support for sparse architectures, particularly mixture-of-experts mode