Awesome GitHub RepositoriesInference Batching

该项目是一个高性能 BERT 嵌入服务和推理服务器，旨在将文本序列映射为固定长度的数值向量。它作为一个机器学习微服务和分布式模型服务器，将请求处理与繁重的计算解耦。 该系统利用 ZeroMQ 消息基础设施在分布式客户端和推理服务器之间提供低延迟通信。它结合了服务器端批处理和 GPU 工作负载扩展，以最大化硬件利用率并管理高请求量。 该平台通过在共享向量空间内为文本和图像生成跨模态嵌入来支持语义搜索基础设施。这实现了跨模态搜索、内容相关性排名以及基于视觉内容与文本描述之间语义对齐的结果重排序。 该服务可以作为可通过 gRPC、HTTP 或 WebSocket 协议访问的弹性微服务进行部署，具有用于处理大数据集的非阻塞双工流。

StreamDiffusion is an interactive generative AI framework and inference engine designed for the low-latency delivery of image and video streams. It provides a real-time Stable Diffusion pipeline for text-to-image and image-to-image generation, enabling the creation of continuous generative image streams with minimized computational delay. The framework optimizes throughput using a pre-computed cache engine and residual-based guidance approximation to reduce the number of required model passes. It further manages GPU load through similarity-based frame skipping, which avoids redundant computat

FlexLLMGen is an inference engine and runtime designed to run large language models on a single GPU by combining weight compression with tensor offloading. It reduces model weight memory usage by approximately 70% through 4-bit quantization, and stores model parameters, attention cache, and hidden states across GPU, CPU, and disk to fit models larger than available GPU memory. The project distinguishes itself through a throughput-oriented batching approach that processes multiple generation requests together in large batches to maximize throughput on a single GPU. It also supports distributed

This project is an AI singing voice conversion system and vocal processor used for training generative voice models and converting vocal recordings or live input into a target voice. It functions as a VITS model trainer and a real-time voice changer that transforms vocal timbre and pitch to change the identity of a singer. The system provides a graphical management dashboard for controlling training hyperparameters and voice conversion presets. It supports low-latency audio streaming for live microphone input and employs pitch estimation to ensure precise matching between source and target vo

KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere

KServe is an open platform for deploying and serving generative and predictive AI models on Kubernetes. It defines inference services as custom resources with declarative YAML specifications, enabling a Kubernetes-native approach to model deployment and lifecycle management. The platform leverages Knative-based serverless scaling for automatic scale-to-zero and revision management, and supports a pluggable serving runtime architecture that maps model formats to containerized execution environments. KServe distinguishes itself through model-aware autoscaling that scales replicas based on token

本项目是一套 MLOps 架构指南和框架，旨在设计并将深度学习系统部署到生产环境。它为模型推理部署、机器学习流水线编排以及生产级机器学习架构的构建提供了结构化的方法。 该项目的特色在于专注于分布式深度学习和边缘 AI 优化。它涵盖了在多个 GPU 上并行化模型训练以处理大规模数据集的方法，并应用了量化和蒸馏等技术来减小嵌入式硬件上的模型体积。 其功能范围还扩展到了监控和可观测性，包括跟踪模型性能、数据漂移和实验指标。此外，它还解决了数据工作流编排、通过对象存储进行数据集版本控制，以及使用自适应批处理和容器化编排来管理高并发推理请求的问题。

exllamav2 是一个高性能推理库，旨在在消费级 GPU 上本地运行大语言模型。它提供了一个 GPU 加速的运行器和量化工具，使模型执行无需依赖基于云的计算服务。 该项目具有一个量化实用程序，可将模型压缩为 2 到 8 位之间的混合比特率，以减少显存需求。它通过一个处理分组请求并对缓存数据进行去重的批处理文本生成器脱颖而出，从而提高了吞吐量。 该库涵盖了广泛的功能，包括用于实时输出的异步 Token 流式传输、用于线性代数运算的自定义 GPU 内核执行，以及用于低延迟访问模型权重的本地内存映射。

exllamav2 是一个高性能推理引擎和框架，用于在消费级 GPU 上本地执行大语言模型。它提供了一个完整的本地模型部署系统，包括专门的推理引擎和模型量化工具。 该项目具有一个多 GPU 推理框架，可将工作负载分配到多个显卡上，以运行超过单个设备内存容量的模型。它包括一个 GPU 模型量化器，能够将模型转换为 2 到 8 位之间的混合精度格式，以平衡内存使用和准确性。 该引擎通过基于批处理的并行推理和异步输出流式传输支持高吞吐量文本生成。这些功能由自定义 CUDA 内核和缓存去重支持，以优化硬件利用率并减少 Token 生成期间的延迟。

这是一个 PyTorch 模型服务框架，旨在通过可扩展的网络端点在生产环境中部署和扩展机器学习模型。它充当高性能推理服务器、优化器和模型生命周期管理器，处理模型加载、请求批处理和硬件加速。 该系统通过先进的编排和优化功能脱颖而出，例如使用执行图将多个模型链接到顺序工作流中，以及采用动态批处理来提高吞吐量和降低延迟。它通过连续批处理和张量并行化为生成式 AI 和大型语言模型提供专门支持。 广泛的功能领域包括跨 NVIDIA、AMD 和 Apple Silicon 等不同硬件的 GPU 资源管理，以及用于注册、版本控制和工作节点扩展的全面模型生命周期管理。它还集成了用于通过 Prometheus 兼容指标跟踪系统健康状况和模型性能的可观测性工具。 该服务器通过用于生命周期控制和运行时参数配置的命令行界面进行管理。

tiny-llm is a large language model inference engine and transformer model implementation. It serves as a quantized model runtime and paged key-value cache manager, providing a specialized inference stack optimized for Apple Silicon. The system distinguishes itself through high-throughput execution techniques, including continuous batching and paged attention. It utilizes a paged memory system to eliminate fragmentation during token generation and employs on-the-fly dequantization of compressed weights to reduce the memory footprint during matrix multiplication. The project covers a broad ran

LitServe 是一个 Python AI 推理服务器框架和大语言模型 (LLM) 服务框架，专为高并发推理而设计。它作为一个分布式 AI 模型服务器和动态批处理推理引擎，提供了构建和托管运行 AI 模型的自定义服务器的工具。 该框架凭借动态批处理请求队列脱颖而出，该队列将单个推理请求分组为单个张量，以最大化 GPU 吞吐量。它支持分布式 GPU 扩展，允许模型工作负载分布在多个硬件加速器上，以平衡计算负载并增加总容量。 该系统提供了一个高级封装接口，将请求预处理和后处理与核心模型执行逻辑解耦。它还包括用于增量交付输出的实时模型流式传输功能，并使用异步事件循环来处理并发网络请求。

LightLLM is a high-performance serving framework for deploying and executing large language models. It functions as a multi-GPU inference engine and server capable of handling dense architectures, mixture-of-experts designs, and multimodal models that process both text and images. The system is distinguished by its specialized support for Mixture-of-Experts models using expert parallelism and fused kernels. It implements structured text generation through deterministic state machines and pushdown automata to enforce precise output formats. To optimize throughput, the framework employs specula

WhisperLive is a real-time speech-to-text server that converts live audio streams into text using Whisper models. It functions as a backend service that receives microphone input via WebSockets and provides incremental transcriptions with word-level timestamps. The system utilizes a GPU-accelerated inference engine and a keyword-boosted transcription API to improve the recognition accuracy of domain-specific jargon, acronyms, and product names. It also includes a speaker diarization tool that clusters audio embeddings to identify and label different participants within a recording. Additiona

Lorax is a GPU-accelerated inference server and multi-adapter engine designed for serving large language models. It functions as a high-throughput system capable of deploying models via Kubernetes and managing the dynamic swapping of Low-Rank Adaptation adapters per request. The server distinguishes itself through multi-adapter dynamic batching, which allows requests using different adapter weights to be processed in a single GPU forward pass. It employs just-in-time adapter loading and weighted adapter merging to maximize throughput and enable multi-tasking without sacrificing performance.

mini-sglang is a collection of tools for large language model inference, serving as an OpenAI-compatible inference server, a memory-efficient prefill engine, and a tensor parallelism runtime. It also functions as a local batch processing engine for offline benchmarking and ablation studies. The project focuses on acceleration and memory management through a KV cache manager that reuses precomputed caches for shared request prefixes. It handles large model workloads by distributing tasks across multiple GPUs and manages peak memory consumption by splitting long input sequences into smaller chu

llm-d is a distributed serving framework designed for large language model inference. It functions as an inference orchestrator and gateway, providing a control plane for deploying model replicas and managing hardware accelerators. The system includes a batch inference scheduler and a cache manager to coordinate request flow and memory utilization. The project is distinguished by a disaggregated serving architecture that separates prefill and decode execution phases across specialized workers to maximize throughput. It employs a hardware-agnostic control plane and tiered cache offloading, mov