7 रिपॉजिटरी
Awesome GitHub RepositoriesServing Frameworks
Platforms and engines designed for the high-throughput deployment and serving of large language models.
Distinct from Large Language Model Optimization: Distinct from general optimization methods: focuses on the serving infrastructure and API layer for LLMs.
Explore 7 awesome GitHub repositories matching artificial intelligence & ml · Serving Frameworks. Refine with filters or upvote what's useful.
Awesome Serving Frameworks GitHub Repositories
AI के साथ बेहतरीन रिपॉजिटरी खोजें।हम AI का उपयोग करके सबसे सटीक रिपॉजिटरी खोजेंगे।
- sgl-project/sglang
sgl-project/sglang
29,079GitHub पर देखेंSglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr
Serves as a production-ready inference engine for large language models with OpenAI-compatible API support.
Pythonattentionblackwellcuda - kvcache-ai/ktransformers
kvcache-ai/ktransformers
17,288GitHub पर देखेंKtransformers is a comprehensive framework designed for the operation, fine-tuning, and serving of large language models. It functions as a heterogeneous inference engine and quantized execution runtime, enabling the deployment of massive models by distributing computational workloads across both CPU and GPU resources. This architecture allows users to bypass local memory constraints, making it possible to run and train models that exceed the capacity of a single device. The project distinguishes itself through specialized support for sparse architectures, particularly mixture-of-experts mode
Integrates high-performance execution kernels into production-ready serving frameworks for hybrid CPU-GPU workloads.
Python - bentoml/openllm
bentoml/OpenLLM
12,115GitHub पर देखेंOpenLLM is a framework for deploying, managing, and scaling open-source large language models
Provides a platform for deploying, managing, and scaling open-source large language models as standardized API endpoints for production applications.
Pythonbentomlfine-tuningllama - huggingface/text-generation-inference
huggingface/text-generation-inference
10,775GitHub पर देखेंText Generation Inference is a production-ready engine designed for the deployment and serving of large language models. It functions as a containerized runtime environment that manages model execution, scales across distributed hardware, and provides high-performance inference capabilities for demanding production environments. The project distinguishes itself through advanced optimization techniques, including continuous batching to maximize hardware utilization and tensor parallelism to shard large models across multiple accelerator cards. It supports efficient inference through custom com
Serves large language models with high-performance infrastructure designed for multi-accelerator deployment.
Pythonbloomdeep-learningfalcon - lmcache/lmcache
LMCache/LMCache
6,909GitHub पर देखेंLMCache is a distributed key-value cache manager and tiering system designed to accelerate large language model inference. It functions as a tiered storage layer that offloads tensors from GPU memory to CPU RAM, local disks, or remote object stores, enabling the reuse of cached prefixes across different inference sessions and serving engines. The system differentiates itself through a disaggregated prefill-decode model, which separates prompt processing from token generation by transferring caches between distributed compute nodes. It utilizes peer-to-peer orchestration to share and retrieve
Links the cache management layer to external serving frameworks to enable the reuse of cached tensors across instances.
Pythonamdcudafast - clearml/clearml
clearml/clearml
6,740GitHub पर देखेंClearML is a comprehensive MLOps platform designed to manage the end-to-end machine learning lifecycle, from initial experimentation to production deployment. It provides a suite of integrated tools including a pipeline orchestrator for automating workflows, an experiment tracking tool for logging hyperparameters and metrics, and a metadata-driven data versioning system for managing large-scale datasets and model artifacts. The platform is distinguished by its advanced compute management and serving capabilities. It features a GPU compute manager that supports fractional resource slicing and
Provides a deployment engine for hosting large language models with API gateways and RAG support.
Python - lightning-ai/litserve
Lightning-AI/LitServe
3,894GitHub पर देखेंLitServe एक Python AI अनुमान सर्वर फ्रेमवर्क और LLM सर्विंग फ्रेमवर्क है जिसे उच्च-कॉनकरेंसी अनुमान के लिए डिज़ाइन किया गया है। यह एक वितरित AI मॉडल सर्वर और डायनामिक बैचिंग अनुमान इंजन के रूप में कार्य करता है, जो AI मॉडल चलाने वाले कस्टम सर्वर बनाने और होस्ट करने के लिए टूल प्रदान करता है। यह फ्रेमवर्क एक डायनामिक-बैचिंग रिक्वेस्ट कतार के माध्यम से खुद को अलग करता है जो GPU थ्रूपुट को अधिकतम करने के लिए व्यक्तिगत अनुमान अनुरोधों को एकल टेंसर में समूहित करता है। यह वितरित GPU स्केलिंग का समर्थन करता है, जिससे मॉडल वर्कलोड को कंप्यूट लोड को संतुलित करने और कुल क्षमता बढ़ाने के लिए कई हार्डवेयर एक्सेलेरेटर्स में फैलाया जा सकता है। सिस्टम एक उच्च-स्तरीय रैपर इंटरफ़ेस प्रदान करता है जो मुख्य मॉडल निष्पादन लॉजिक से अनुरोध प्रीप्रोसेसिंग और पोस्टप्रोसेसिंग को अलग करता है। इसमें आउटपुट को क्रमिक रूप से डिलीवर करने के लिए रीयल-टाइम मॉडल स्ट्रीमिंग के लिए क्षमताएं भी शामिल हैं और समवर्ती नेटवर्क अनुरोधों को संभालने के लिए एक एसिंक्रोनस इवेंट लूप का उपयोग करता है।
Provides a high-throughput deployment and serving framework specifically for large language models.
Python
