9 个仓库
Awesome GitHub RepositoriesDynamic Inference Batching
Combines short requests into batches and splits long sequences across GPUs to balance throughput and latency.
Distinct from Request Batching: Distinct from Request Batching: focuses on dynamic batching for inference workloads with sequence splitting, not general data operation batching.
Explore 9 awesome GitHub repositories matching data & databases · Dynamic Inference Batching. Refine with filters or upvote what's useful.
Awesome Dynamic Inference Batching GitHub Repositories
用 AI 发现最棒的仓库。我们将通过 AI 为您搜索最匹配的仓库。
- wang-xinyu/tensorrtx
wang-xinyu/tensorrtx
7,802在 GitHub 上查看tensorrtx is a computer vision inference engine and model implementation library designed for graphics processor acceleration. It provides a framework for optimizing deep learning models through a GPU inference optimizer, a deep learning model converter for transforming weights from frameworks like TensorFlow and PyTorch, and a custom plugin library to implement operations not natively supported by the TensorRT API. The project distinguishes itself through a comprehensive collection of pre-defined network implementations, ranging from various YOLO versions and DETR transformers for object det
Implements dynamic batching for inference workloads to optimize the balance between throughput and latency.
C++arcfacecrnndetr - infrasys-ai/aiinfra
Infrasys-AI/AIInfra
7,414在 GitHub 上查看Combines short requests into batches and splits long sequences across GPUs for balanced throughput.
Jupyter Notebookaiinfraaisystem - nvidia/isaac-gr00t
NVIDIA/Isaac-GR00T
6,222在 GitHub 上查看Combines dynamic batching and concurrent execution to maximize hardware utilization during model serving.
Jupyter Notebook - imoneoi/openchat
imoneoi/openchat
5,481在 GitHub 上查看OpenChat 是一个用于训练、微调和部署大语言模型的框架，针对对话和数学推理任务进行了优化。它提供了从训练流水线、部署栈到基于 Web 的聊天界面的全生命周期管理。 该项目专注于在消费级硬件上实现高性能模型执行，无需企业级加速器。它包含一个生产就绪的推理服务器，实现了 OpenAI 聊天补全协议，并利用动态请求批处理来优化硬件吞吐量。 该系统涵盖了整个操作工作流，包括数据集分词、通过无填充训练（padding-free training）进行模型微调以及强化学习。它还扩展到支持基于密钥认证的 API 托管，并提供用于实时人机交互的图形用户界面。
Uses dynamic request batching to group multiple API requests into a single inference pass for higher throughput.
Python - ztxz16/fastllm
ztxz16/fastllm
4,779在 GitHub 上查看fastllm 是一套用于模型权重转换、混合专家 (MoE) 运行时和张量并行的专用软件组件。它提供了一个兼容 OpenAI 的 API 服务器，通过标准化的请求格式公开大语言模型功能。 该项目具有一个张量并行框架，可将计算工作负载拆分到多个 GPU 上以加速执行。它包含一个针对混合专家架构优化的专用运行时，以及一个将模型权重转换为低精度格式以减少内存使用并提高吞吐量的量化工具。 系统涵盖了分布式推理的高级工作流，包括设备映射内存管理、动态批处理和混合模式执行。它还提供了一个用于模型管理和部署配置的命令行界面和终端用户界面。
Groups multiple incoming requests into single execution passes to maximize GPU utilization and reduce token latency.
C++ - tianxiaomo/pytorch-yolov4
Tianxiaomo/pytorch-YOLOv4
4,526在 GitHub 上查看该项目是 YOLOv4 目标检测框架的 PyTorch 实现。它提供了一个用于训练和部署神经网络的系统，该系统可以识别并定位图像和视频流中的多个对象。 该框架包括将训练权重转换为通用格式和硬件特定优化引擎的工具，专门支持 ONNX 和 TensorRT。它具有 TensorRT 推理优化器以减少延迟并提高吞吐量，以及与 NVIDIA DeepStream 流分析管道兼容的模型架构。 该系统涵盖了使用马赛克数据增强的模型训练，并支持跨多个推理引擎的实时目标检测。它提供了用于模型转换为 TensorFlow 的实用程序，并支持用于部署的静态和动态批处理配置。
Supports both static and dynamic batch configurations to optimize GPU memory usage and inference throughput.
Pythondarknet2onnxdarknet2pytorchonnx - opennmt/ctranslate2
OpenNMT/CTranslate2
4,319在 GitHub 上查看CTranslate2 is a C++ inference engine and runtime for Transformer models, designed to execute models on both CPU and GPU with optimizations for speed and memory efficiency. It functions as a model format converter, quantization tool, and REST API server, enabling deployment of neural machine translation, automatic speech recognition, and text generation models. The engine distinguishes itself through a suite of runtime optimizations including layer fusion, weight-matrix quantization, batch-by-length grouping, and a caching allocator that reuses GPU memory. It supports tensor-parallel model di
Processes multiple requests in parallel across CPU cores or GPUs, with dynamic memory allocation per batch size.
C++avxavx2cpp - thudm/slime
THUDM/slime
4,259在 GitHub 上查看SLIME is a distributed reinforcement learning framework for large language model post-training that bridges Megatron training with SGLang inference servers. It orchestrates scalable RL loops across GPU clusters, decoupling training and inference into independent processes that communicate over HTTP and NCCL for independent scaling and fault tolerance. The system supports multi-agent reinforcement learning workflows with parallel agent instances, customizable rollout strategies, and personalized agent serving that improves models from prior conversations without disrupting API serving. The fra
Packs variable-length sequences into batches up to a token limit per GPU, preserving per-sample loss while maximizing throughput.
Python - lightning-ai/litserve
Lightning-AI/LitServe
3,894在 GitHub 上查看LitServe 是一个 Python AI 推理服务器框架和大语言模型 (LLM) 服务框架，专为高并发推理而设计。它作为一个分布式 AI 模型服务器和动态批处理推理引擎，提供了构建和托管运行 AI 模型的自定义服务器的工具。 该框架凭借动态批处理请求队列脱颖而出，该队列将单个推理请求分组为单个张量，以最大化 GPU 吞吐量。它支持分布式 GPU 扩展，允许模型工作负载分布在多个硬件加速器上，以平衡计算负载并增加总容量。 该系统提供了一个高级封装接口，将请求预处理和后处理与核心模型执行逻辑解耦。它还包括用于增量交付输出的实时模型流式传输功能，并使用异步事件循环来处理并发网络请求。
Implements a dynamic-batching request queue to maximize GPU throughput by grouping individual requests.
Python
