6 个仓库
Awesome GitHub RepositoriesMulti-GPU Inference Runtimes
Runtimes that distribute model execution across multiple GPUs using tensor parallelism to handle large models.
Distinct from Model Inference Runtimes: Distinct from Model Inference Runtimes: focuses on multi-GPU distribution via tensor parallelism, not single-device execution.
Explore 6 awesome GitHub repositories matching artificial intelligence & ml · Multi-GPU Inference Runtimes. Refine with filters or upvote what's useful.
Awesome Multi-GPU Inference Runtimes GitHub Repositories
用 AI 发现最棒的仓库。我们将通过 AI 为您搜索最匹配的仓库。
- mistralai/mistral-src
mistralai/mistral-src
10,821在 GitHub 上查看该项目是一个大语言模型推理库和框架，旨在运行用于文本生成、问题解决和编码辅助的模型。它包括一个用于处理图像和文本组合输入的多模态框架，以及一个基于模型推理执行外部工具的工具调用实现。 该系统具有分布式 GPU 推理引擎，可将大型模型工作负载分散到多个图形处理器上，以提高处理速度并满足内存需求。它还通过预打包的镜像和依赖项提供容器化模型部署，以便在隔离环境中运行推理引擎。 该库涵盖了一系列功能，包括多模态输入分析、函数调用集成，以及用于预测缺失代码段的“中间填充”（fill-in-the-middle）编码。它还支持通过命令行界面进行交互式模型聊天，以维持对话会话。
Implements a runtime that distributes model execution across multiple GPUs using tensor parallelism to handle large model weights.
Jupyter Notebook - qqwweee/keras-yolo3
qqwweee/keras-yolo3
7,116在 GitHub 上查看This project is an object detection framework implementing the YOLOv3 architecture using Keras and TensorFlow. It functions as a deep learning vision model and computer vision toolset designed to locate and classify multiple entities within images and video streams using bounding boxes. The system includes a multi-GPU inference engine to distribute computational loads across several graphics processing units. It also provides a pipeline for creating custom object detectors by retraining pre-trained weights on annotated datasets to recognize user-defined object classes. The framework covers m
Distributes the computational load of deep learning predictions across multiple graphics processing units.
Python - thudm/cogvlm
THUDM/CogVLM
6,742在 GitHub 上查看CogVLM is a multimodal large language model designed to integrate visual and textual data for reasoning about images and generating natural language. It functions as a visual question answering system that analyzes image content to provide detailed descriptions or answer specific questions. The project includes a visual grounding model capable of mapping text descriptions to precise bounding box coordinates within an image. It also features a vision-based automation agent that analyzes screen captures to generate execution plans and interaction coordinates for software interfaces. The system
Optimizes GPU inference through weight quantization and distributed execution across multiple graphics processors.
Python - pytorch-labs/gpt-fast
pytorch-labs/gpt-fast
6,225在 GitHub 上查看gpt-fast 是一个 PyTorch Transformer 推理引擎，专为低延迟文本生成而设计。它作为一个分布式 GPU 推理库、量化模型运行器和推测解码框架。 该系统利用推测解码工作流，其中小型草稿模型预测 Token 序列，由较大模型进行验证以加速生成。它支持量化模型执行以减少内存占用，并实现张量并行以跨多个 GPU 分割计算。 该项目包括一个标准化的评估工具，用于衡量 Transformer 模型的准确性和性能。它通过键值（KV）缓存管理和原生 PyTorch 张量操作来管理推理效率。
Acts as a multi-GPU inference runtime that utilizes tensor parallelism for large transformer models.
Python - turboderp-org/exllamav2
turboderp-org/exllamav2
4,552在 GitHub 上查看exllamav2 是一个高性能推理引擎和框架，用于在消费级 GPU 上本地执行大语言模型。它提供了一个完整的本地模型部署系统，包括专门的推理引擎和模型量化工具。 该项目具有一个多 GPU 推理框架，可将工作负载分配到多个显卡上，以运行超过单个设备内存容量的模型。它包括一个 GPU 模型量化器，能够将模型转换为 2 到 8 位之间的混合精度格式，以平衡内存使用和准确性。 该引擎通过基于批处理的并行推理和异步输出流式传输支持高吞吐量文本生成。这些功能由自定义 CUDA 内核和缓存去重支持，以优化硬件利用率并减少 Token 生成期间的延迟。
Provides a framework for distributing large language model workloads across multiple GPUs to bypass memory limits.
Python - opennmt/ctranslate2
OpenNMT/CTranslate2
4,319在 GitHub 上查看CTranslate2 is a C++ inference engine and runtime for Transformer models, designed to execute models on both CPU and GPU with optimizations for speed and memory efficiency. It functions as a model format converter, quantization tool, and REST API server, enabling deployment of neural machine translation, automatic speech recognition, and text generation models. The engine distinguishes itself through a suite of runtime optimizations including layer fusion, weight-matrix quantization, batch-by-length grouping, and a caching allocator that reuses GPU memory. It supports tensor-parallel model di
Distributing model execution across multiple GPUs using tensor parallelism to handle large models that exceed single-device memory.
C++avxavx2cpp
