高吞吐量 LLM 推理服务器
For 用于大规模部署 LLM 的推理服务器, the strongest matches are internlm/lmdeploy (lmdeploy is a high-performance inference engine and deployment framework), sgl-project/mini-sglang (mini-sglang is an OpenAI-compatible LLM inference server with tensor) and nvidia/tensorrt-llm (TensorRT-LLM is NVIDIA’s platform for compiling, optimizing, and serving). predibase/lorax and vllm-project/vllm round out the shortlist. Each is ranked by relevance to your query, popularity and recent activity.
旨在以优化延迟和高吞吐量服务大语言模型的生产级软件框架。
高吞吐量 LLM 推理服务器
用 AI 发现最棒的仓库。我们将通过 AI 为您搜索最匹配的仓库。
- internlm/lmdeploy
InternLM/lmdeploy
7,903在 GitHub 上查看lmdeploy is a high-performance inference engine and deployment framework for large language models and vision models. It functions as a multi-modal model server and compression toolkit designed to serve models with high throughput and low latency. The system enables the distribution of model services across multiple machines using request-based load balancing and tensor parallelism. It includes specialized tools for model quantization and compression to reduce the memory footprint of weights and caches. The framework covers broad capability areas including production deployment, distributed
lmdeploy is a high-performance inference engine and deployment framework for LLMs and vision models, supporting distributed tensor parallelism, continuous batching, model quantization, and a production-grade serving API — exactly what this search demands.
PythonContinuous Batching StrategiesHigh-Throughput Model ServingTensor Parallelism - sgl-project/mini-sglang
sgl-project/mini-sglang
3,514在 GitHub 上查看mini-sglang is a collection of tools for large language model inference, serving as an OpenAI-compatible inference server, a memory-efficient prefill engine, and a tensor parallelism runtime. It also functions as a local batch processing engine for offline benchmarking and ablation studies. The project focuses on acceleration and memory management through a KV cache manager that reuses precomputed caches for shared request prefixes. It handles large model workloads by distributing tasks across multiple GPUs and manages peak memory consumption by splitting long input sequences into smaller chu
mini-sglang is an OpenAI-compatible LLM inference server with tensor parallelism, prefix caching, and KV cache management for high throughput, directly meeting the need for a production-grade serving solution.
PythonTensor-Parallel Inference DistributionsTensor Parallelism - nvidia/tensorrt-llm
NVIDIA/TensorRT-LLM
12,913在 GitHub 上查看TensorRT-LLM is a platform and toolkit designed for compiling, optimizing, and serving transformer-based models on accelerated hardware. It functions as a framework that transforms machine learning models into efficient execution graphs, providing an engine to refine these models for specific hardware to maximize throughput and minimize latency during text generation. The project distinguishes itself through advanced execution strategies that manage the entire inference pipeline. It utilizes kernel-level fusion and static graph execution to optimize mathematical operations and computational f
TensorRT-LLM is NVIDIA’s platform for compiling, optimizing, and serving large transformer models on GPUs, delivering high throughput, low latency, and advanced batching strategies—exactly what you need for production-grade LLM inference.
PythonGPU-AcceleratedHigh-Throughput Model Serving - predibase/lorax
predibase/lorax
3,724在 GitHub 上查看Lorax is a GPU-accelerated inference server and multi-adapter engine designed for serving large language models. It functions as a high-throughput system capable of deploying models via Kubernetes and managing the dynamic swapping of Low-Rank Adaptation adapters per request. The server distinguishes itself through multi-adapter dynamic batching, which allows requests using different adapter weights to be processed in a single GPU forward pass. It employs just-in-time adapter loading and weighted adapter merging to maximize throughput and enable multi-tasking without sacrificing performance.
Lorax is a GPU-accelerated LLM inference server with dynamic batching, multi‑GPU distribution, Kubernetes orchestration, and a REST API—exactly the high‑throughput, production‑grade setup this search targets.
PythonGPU-Accelerated InferenceTensor-Parallel Inference DistributionsSpeculative Decoding Strategies - vllm-project/vllm
vllm-project/vllm
83,048在 GitHub 上查看vLLM is a high-throughput inference engine designed for the efficient serving and execution of large language models. It functions as a production-ready distributed model server, providing standard API protocols for online serving while also supporting offline batch processing. The system is built to maximize token generation speed and memory efficiency, enabling both large-scale cloud deployments and local execution on personal hardware. The project distinguishes itself through advanced memory management and request scheduling techniques, most notably its use of non-contiguous key-value cach
vLLM is a production-grade, high-throughput inference engine purpose-built for serving large language models with efficient memory management and continuous batching, making it an ideal match for this search.
PythonContinuous Batching StrategiesHigh-Throughput Model Serving - huggingface/text-generation-inference
huggingface/text-generation-inference
10,775在 GitHub 上查看Text Generation Inference is a production-ready engine designed for the deployment and serving of large language models. It functions as a containerized runtime environment that manages model execution, scales across distributed hardware, and provides high-performance inference capabilities for demanding production environments. The project distinguishes itself through advanced optimization techniques, including continuous batching to maximize hardware utilization and tensor parallelism to shard large models across multiple accelerator cards. It supports efficient inference through custom com
Text Generation Inference is a production-ready LLM serving engine with continuous batching, tensor parallelism, and high-throughput inference, directly matching your need for a scalable, GPU-accelerated inference server with a REST/streaming API.
PythonContinuous Batching StrategiesTensor Parallelism - geeeekexplorer/nano-vllm
GeeeekExplorer/nano-vllm
11,745在 GitHub 上查看Nano-vllm is a high-performance inference engine designed for executing large language models locally. It functions as a specialized runtime that prioritizes accelerated token generation and efficient hardware utilization for text generation tasks. The project distinguishes itself through a comprehensive suite of optimization techniques, including a graph compilation engine that transforms neural network operations into pre-compiled execution plans. It also incorporates a tensor parallelism framework to distribute model weights across multiple hardware accelerators, effectively reducing memor
Nano-vllm is a high-performance inference engine for LLMs that supports continuous batching and tensor parallelism, directly targeting the category, though its focus on local execution and missing mention of a REST/streaming API means it may need additional setup for production-grade serving.
PythonContinuous Batching StrategiesHigh-Throughput Model ServingTensor Parallelism Frameworks - modeltc/lightllm
ModelTC/LightLLM
3,901在 GitHub 上查看LightLLM is a high-performance serving framework for deploying and executing large language models. It functions as a multi-GPU inference engine and server capable of handling dense architectures, mixture-of-experts designs, and multimodal models that process both text and images. The system is distinguished by its specialized support for Mixture-of-Experts models using expert parallelism and fused kernels. It implements structured text generation through deterministic state machines and pushdown automata to enforce precise output formats. To optimize throughput, the framework employs specula
LightLLM is a high-performance multi-GPU inference server built for LLMs, offering expert parallelism, continuous batching, and REST/streaming APIs, which directly matches your need for production-grade, efficient LLM serving.
PythonHybrid Model ParallelismTensor ParallelismSpeculative Decoding Strategies - qwenlm/qwen
QwenLM/Qwen
21,294在 GitHub 上查看Qwen is a comprehensive framework for large language model development, serving, and deployment. It provides a complete ecosystem for transformer-based sequence modeling, offering base models alongside specialized tools for instruction-tuned alignment, fine-tuning, and long-context inference. The project is designed to support both research and production environments, enabling users to train, optimize, and host generative models locally or across distributed hardware. The framework distinguishes itself through its focus on high-performance serving and extensibility. It features a high-perfor
Qwen is an LLM framework that includes a high-performance inference server with support for continuous batching, GPU acceleration, multi-GPU deployment, and streaming APIs, making it a good fit for production-grade high-throughput serving.
PythonHigh-Throughput Model ServingTensor Parallelism - sgl-project/sglang
sgl-project/sglang
29,079在 GitHub 上查看Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr
SGLang is a high-performance inference engine and serving system purpose-built for large language and multimodal models, offering an OpenAI-compatible API, GPU acceleration via CUDA, and production-grade deployment features — directly matching the need for high-throughput LLM serving with efficient resource use.
PythonContinuous Batching StrategiesHigh-Throughput Model ServingModel Parallelism Strategies - triton-inference-server/server
triton-inference-server/server
10,768在 GitHub 上查看Triton Inference Server is a high-performance server designed to deploy machine learning models from multiple frameworks across GPUs and CPUs. It functions as a hardware-accelerated inference engine and a gRPC inference gateway, providing a standardized communication layer for transmitting binary tensor data with low latency. The system acts as a multi-framework model orchestrator, allowing users to link multiple AI models into ensembles and scripts to create complex inference pipelines. It also serves as a model lifecycle manager, providing controls to load, unload, and monitor the performan
Triton Inference Server is a production-grade platform for deploying ML models with GPU acceleration, dynamic batching, and REST/gRPC APIs, making it well-suited for high-throughput LLM serving, though it is a general-purpose server rather than LLM-optimized.
PythonHigh-Throughput Model ServingHardware-Accelerated Inference - vllm-project/vllm-omni
vllm-project/vllm-omni
2,776在 GitHub 上查看vllm-omni is a high-throughput serving engine and distributed inference framework designed for omni-modal models. It serves as a multi-modal model API server capable of generating text, image, video, and audio data, providing a standardized interface for remote client access. The system features a non-autoregressive generation engine for parallel media production and a robot policy inference server that acts as a real-time communication bridge to robotic hardware using specialized protocols. It supports hybrid execution models that combine sequential token generation with parallelized media g
A production-grade inference engine from the vLLM ecosystem that serves LLMs (among other modalities) with continuous batching, GPU acceleration, model parallelism, and an OpenAI-compatible API — it meets all the required capabilities, though its primary focus on omni-modal models goes beyond plain LLM serving.
PythonContinuous Batching StrategiesTensor Parallelism - tiiny-ai/powerinfer
Tiiny-AI/PowerInfer
8,714在 GitHub 上查看PowerInfer is a high-performance local large language model inference engine and sparse inference framework. It provides a runtime for executing models on consumer-grade hardware, utilizing a GPU acceleration backend to optimize tensor operations for graphics processors. The system distinguishes itself through a sparse inference framework that increases generation speed by skipping computations based on activation sparsity in model weights. It includes a GGUF model converter for transforming weights and metadata into a unified binary format, as well as an OpenAI API compatible server for inte
PowerInfer is a high-performance LLM inference engine with GPU acceleration, continuous batching, and an OpenAI-compatible API server, making it a solid fit for high-throughput local inference, though it is optimized more for consumer-grade hardware than large-scale production deployments.
C++GPU-Accelerated InferenceOpenAI-Compatible Inference Servers - zai-org/chatglm3
zai-org/ChatGLM3
13,764在 GitHub 上查看ChatGLM3 is a comprehensive framework for deploying, fine-tuning, and serving large language models. It functions as a high-performance inference engine designed to support conversational AI, enabling developers to build interactive agents capable of multi-turn dialogue, autonomous code execution, and structured tool invocation. The project distinguishes itself through its focus on hardware-agnostic deployment and resource optimization. It supports distributed model parallelism across multiple graphics cards, paged key-value caching for concurrent request processing, and weight quantization t
Zai-org/chatglm3 is a high-performance inference engine and deployment framework for large language models, supporting distributed model parallelism, paged key-value caching for continuous batching, GPU acceleration, and quantization, which squarely fits the need for a production-grade LLM inference server with efficient resource utilization.
PythonModel Parallelism StrategiesHardware-Accelerated Inference - thu-pacman/chitu
thu-pacman/chitu
3,265在 GitHub 上查看Chitu is a distributed serving platform and orchestrator for large language model inference. It functions as a compute manager designed to deploy and scale model workloads across diverse hardware architectures, including GPUs, CPUs, and heterogeneous hardware clusters. The platform enables model deployment across a wide range of targets, including NVIDIA GPUs, regional chipsets, and legacy hardware. It manages the execution of models across these varying environments to increase available computing capacity and optimize resource utilization. The system includes capabilities for distributed i
Chitu is a distributed serving platform and orchestrator for LLM inference that manages deployment and scaling across heterogeneous hardware, which fits the search for an LLM inference server, though its orchestration focus may differ from a streamlined server like vLLM.
PythonHigh-Throughput Model Serving - xorbitsai/inference
xorbitsai/inference
9,358在 GitHub 上查看This project is a platform for the deployment of open source large language and multimodal models. It provides a unified interface to serve text, image, and speech models across local or cloud hardware. The system enables distributed AI inference by orchestrating model workloads across multiple nodes and devices. It includes a unified API adapter layer to standardize inputs and outputs, as well as tools for multimodal chat and structural image generation. The platform covers a broad capability surface including request batching for throughput optimization, dynamic model loading, and integrat
Xorbits Inference is a unified serving platform for LLMs and multimodal models that supports distributed inference across nodes, request batching for throughput, and OpenAI-compatible APIs, directly meeting the need for a high-throughput LLM inference server with efficient resource utilization.
PythonHardware-Accelerated Inference - zai-org/chatglm-6b
zai-org/ChatGLM-6B
41,039在 GitHub 上查看ChatGLM-6B is a generative AI inference engine designed for local execution of transformer-based language models. It provides a comprehensive runtime environment that allows users to load and run pre-trained neural network weights directly on their own hardware, ensuring data privacy and independence from external cloud services. The project distinguishes itself through a hardware-agnostic execution backend that supports deployment across diverse environments, including standard processors, Apple Silicon, and multi-GPU configurations. It incorporates advanced optimization techniques such as w
The ChatGLM-6B inference engine lets you run transformer-based LLMs locally with multi-GPU and hardware-agnostic support, making it the right category of tool for an LLM inference server, though it may lack explicit high‑throughput batching and REST/streaming API features that a production‑grade server would typically include.
PythonTensor Parallelism - ggml-org/llama.cpp
ggml-org/llama.cpp
116,799在 GitHub 上查看Llama.cpp is an inference engine designed for the local execution of text-based and multimodal language models on consumer hardware. It provides a core environment for running models that process both text and image inputs, utilizing hardware-accelerated backends to optimize performance across diverse CPU and GPU architectures. The project distinguishes itself by offering a lightweight HTTP server that adheres to standard API specifications, enabling chat completion, embeddings, and reranking services. It includes a suite of tools for model quantization and conversion, which reduces memory us
llama.cpp is an LLM inference engine with a lightweight HTTP server that supports GPU acceleration and model quantization, making it a genuine inference server; however, its focus on local/consumer hardware means it may not match the high-throughput production-grade optimization this search targets.
C++Hardware Abstraction LayersText-Only Inference EnginesMultimodal Inference Engines - mudler/localai
mudler/LocalAI
46,889在 GitHub 上查看LocalAI is a self-hosted inference server that enables the execution of machine learning models directly on local hardware. By providing a unified interface for text, image, and audio processing, it allows users to maintain full control over data privacy and infrastructure costs while eliminating dependencies on external network services. The platform functions as an API gateway that mimics standard cloud-based artificial intelligence interfaces, allowing existing applications to integrate local models as drop-in replacements. It utilizes a container-based architecture to package runtimes and
LocalAI is a self-hosted inference server that provides a unified API for running LLMs and other models locally, which fits the category of an LLM inference server, though it emphasizes local deployment and data privacy over the specific high-throughput optimizations you may need for production at scale.
GoInference ServersLocal Inference EnginesLocal Model Serving - ollama/ollama
ollama/ollama
174,300在 GitHub 上查看Ollama provides a framework for running and managing local machine learning models. It includes a command-line interface for model lifecycle management, such as creation, embedding generation, and configuration, alongside a stable API for programmatic interaction across multiple programming languages. The platform supports the import of models and adapters in various formats, including GGUF and Safetensors. Users can define custom model behaviors, prompt templates, and system messages through a configuration file format. It also offers tools for fine-tuning models with LoRA adapters and apply
Ollama is a local LLM inference server with a REST and streaming API that supports GPU acceleration and continuous batching via its llama.cpp backend, making it a solid match for high-throughput serving, though it may prioritize ease of deployment over the most aggressive production optimizations.
GoAI Integration APIsAI Agent FrameworksAI and Machine Learning - lm-sys/fastchat
lm-sys/FastChat
39,472在 GitHub 上查看FastChat is a training and serving platform for large language models that provides an integrated toolkit for fine-tuning, hosting, and benchmarking chatbots. It functions as an inference server capable of hosting multiple models and exposing them via a standardized API for chat applications. The platform distinguishes itself through a distributed model controller that manages worker nodes and routes requests across a hardware-agnostic inference layer supporting various accelerators. It includes a dedicated evaluation framework for assessing model quality using automated judges, multi-turn di
FastChat functions as an inference server that can host multiple LLMs and expose them via a standardized chat API, which matches your need for an LLM serving solution; however, it is designed as a general training-and-serving platform rather than being exclusively optimized for high-throughput production serving with continuous batching.
PythonLarge Language Model ServingChatbot Hosting ServicesDistributed Model Orchestration - mlc-ai/mlc-llm
mlc-ai/mlc-llm
22,057在 GitHub 上查看MLC LLM is a machine learning compiler and inference engine designed to execute large language models locally across diverse hardware platforms, including desktop, mobile, and web environments. By utilizing machine learning compilation, the project transforms high-level model definitions into specialized, hardware-specific binary libraries. This process optimizes model weights and generates compute kernels tailored to the unique memory and processing characteristics of target graphics and mobile hardware. The engine distinguishes itself by providing a unified runtime abstraction that enables
MLC-LLM is a machine learning compiler and inference engine that can serve LLMs via a local API, making it a valid inference server, though its focus on cross-platform compilation rather than pure throughput optimization means it fits the search but is not the most production-oriented flagship in the space.
PythonLocal Inference EnginesLocal Language Model ExecutionLocal Model Inference Servers - xusenlinzy/api-for-open-llm
xusenlinzy/api-for-open-llm
2,460在 GitHub 上查看This project provides a unified server environment and gateway for hosting and executing open-source large language models on private infrastructure. It functions as a standardized interface that exposes locally deployed models through widely-adopted API protocols, allowing existing applications to interact with them without requiring code modifications. The platform distinguishes itself by acting as a compatibility layer that translates standard REST requests into model-specific execution calls. It supports advanced interaction patterns including real-time token streaming, function calling f
This repository wraps multiple open-source LLMs behind an OpenAI-compatible API, so it is an LLM inference server at its core, but it does not highlight the high-throughput optimizations (continuous batching, model parallelism) you’d expect for production-grade serving.
PythonOpenAI-Compatible API Servers - go-skynet/localai
go-skynet/LocalAI
47,157在 GitHub 上查看LocalAI is a local generative AI platform and inference engine designed to host large language, vision, and audio models on private hardware. It functions as an API compatible gateway that mimics proprietary service endpoints, allowing existing third-party software to integrate with a self-hosted backend. The platform distinguishes itself as a distributed AI model orchestrator, capable of scaling inference across machine clusters using VRAM-aware routing and hardware coordination. It provides a unified interface for diverse open-source backends and supports self-hosted RAG infrastructure thro
LocalAI is a local generative AI inference engine and API gateway that can serve LLMs across clusters with VRAM-aware routing, making it an LLM inference server in the right category; while it supports GPU acceleration and a REST API, its local-first design may not be as aggressively optimized for high-throughput serving as purpose-built solutions like vLLM or TGI.
GoOpenAI-Compatible APIsOpenAI-Compatible EndpointsAI-Powered Image and Video Generation
