Awesome GitHub RepositoriesServing Frameworks

Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr

Ktransformers is a comprehensive framework designed for the operation, fine-tuning, and serving of large language models. It functions as a heterogeneous inference engine and quantized execution runtime, enabling the deployment of massive models by distributing computational workloads across both CPU and GPU resources. This architecture allows users to bypass local memory constraints, making it possible to run and train models that exceed the capacity of a single device. The project distinguishes itself through specialized support for sparse architectures, particularly mixture-of-experts mode

OpenLLM is a framework for deploying, managing, and scaling open-source large language models

Text Generation Inference is a production-ready engine designed for the deployment and serving of large language models. It functions as a containerized runtime environment that manages model execution, scales across distributed hardware, and provides high-performance inference capabilities for demanding production environments. The project distinguishes itself through advanced optimization techniques, including continuous batching to maximize hardware utilization and tensor parallelism to shard large models across multiple accelerator cards. It supports efficient inference through custom com

LMCache is a distributed key-value cache manager and tiering system designed to accelerate large language model inference. It functions as a tiered storage layer that offloads tensors from GPU memory to CPU RAM, local disks, or remote object stores, enabling the reuse of cached prefixes across different inference sessions and serving engines. The system differentiates itself through a disaggregated prefill-decode model, which separates prompt processing from token generation by transferring caches between distributed compute nodes. It utilizes peer-to-peer orchestration to share and retrieve

ClearML is a comprehensive MLOps platform designed to manage the end-to-end machine learning lifecycle, from initial experimentation to production deployment. It provides a suite of integrated tools including a pipeline orchestrator for automating workflows, an experiment tracking tool for logging hyperparameters and metrics, and a metadata-driven data versioning system for managing large-scale datasets and model artifacts. The platform is distinguished by its advanced compute management and serving capabilities. It features a GPU compute manager that supports fractional resource slicing and

LitServe هو إطار عمل لخادم استدلال الذكاء الاصطناعي بلغة Python وإطار عمل لخدمة النماذج اللغوية الكبيرة (LLM) مصمم للاستدلال عالي التزامن. يعمل كخادم نماذج ذكاء اصطناعي موزع ومحرك استدلال مجمع ديناميكياً، مما يوفر الأدوات لبناء واستضافة خوادم مخصصة تشغل نماذج الذكاء الاصطناعي. يتميز إطار العمل بطابور طلبات مجمع ديناميكياً يجمع طلبات الاستدلال الفردية في موترات (tensors) واحدة لزيادة إنتاجية GPU. يدعم توسيع نطاق GPU الموزع، مما يسمح بتوزيع أعباء عمل النماذج عبر مسرعات أجهزة متعددة لموازنة أحمال الحوسبة وزيادة السعة الإجمالية. يوفر النظام واجهة غلاف عالية المستوى تفصل المعالجة المسبقة للطلب والمعالجة اللاحقة عن منطق تنفيذ النموذج الأساسي. كما يتضمن إمكانات لبث النماذج في الوقت الفعلي لتقديم المخرجات بشكل تزايدي ويستخدم حلقة أحداث غير متزامنة للتعامل مع طلبات الشبكة المتزامنة.