Awesome GitHub RepositoriesPrefill-Decode Disaggregation

Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr

FlashMLA is an LLM attention kernel library and inference acceleration library providing a collection of high-performance CUDA kernels. It implements multi-head latent attention mechanisms designed to reduce memory overhead and increase throughput during the forward and backward passes of large language model inference. The library utilizes quantized cache attention kernels to improve computation efficiency across both sparse and dense token processing. It specifically optimizes the prefill and decoding phases of model inference through these latent attention implementations. The project cov

LMCache is a distributed key-value cache manager and tiering system designed to accelerate large language model inference. It functions as a tiered storage layer that offloads tensors from GPU memory to CPU RAM, local disks, or remote object stores, enabling the reuse of cached prefixes across different inference sessions and serving engines. The system differentiates itself through a disaggregated prefill-decode model, which separates prompt processing from token generation by transferring caches between distributed compute nodes. It utilizes peer-to-peer orchestration to share and retrieve

Dynamo is a distributed inference orchestration platform designed for large language models. It functions as a system to coordinate prefill and decode phases across GPU nodes, utilizing a multi-backend runtime adapter to connect engines like vLLM and TensorRT-LLM through a unified block-oriented memory interface. An OpenAI-compatible API server provides the frontend for integration with existing tools and clients. The project is distinguished by its disaggregated serving architecture, which separates prompt processing and token generation onto independent GPU pools to optimize throughput and

Mooncake एक डिसएग्रीगेटेड लार्ज लैंग्वेज मॉडल सर्विंग प्लेटफ़ॉर्म और डिस्ट्रीब्यूटेड की-वैल्यू स्टोर है जिसे हाई-परफ़ॉर्मेंस इन्फरेंस इंफ्रास्ट्रक्चर के लिए डिज़ाइन किया गया है। यह एक GPU मेमोरी ऑर्केस्ट्रेटर और KV कैश मैनेजमेंट सिस्टम के रूप में कार्य करता है जो इन्फरेंस को तेज़ करने के लिए क्लस्टर्स में की-वैल्यू कैश को पूल और ट्रांसफर करता है। यह सिस्टम इन्फरेंस के प्रीफ़िल और डिकोड चरणों को अलग-अलग हार्डवेयर क्लस्टर्स में विभाजित करके संसाधन उपयोग को ऑप्टिमाइज़ करता है। यह कंप्यूट नोड्स के बीच डेटा को मूव करने के लिए ज़ीरो-कॉपी ट्रांसफ़र के साथ एक हाई-परफ़ॉर्मेंस RDMA डिस्ट्रीब्यूटेड कैश का उपयोग करता है, जिससे लेटेंसी और ओवरहेड को कम करने के लिए CPU को बायपास किया जाता है। प्लेटफ़ॉर्म डिस्ट्रीब्यूटेड मेमोरी पूलिंग, CXL के माध्यम से एक्सेलेरेटर मेमोरी रूटिंग, और SSDs में मल्टी-टियर स्टोरेज ऑफ़लोडिंग सहित व्यापक क्षमताओं को कवर करता है। यह मेटाडेटा कोऑर्डिनेशन सर्विसेज़ के माध्यम से क्लस्टर स्टेट को मैनेज करता है और लीज़-बेस्ड ऑब्जेक्ट प्रोटेक्शन व वॉटरमार्क-बेस्ड कैश इविक्शन के माध्यम से संसाधन गवर्नेंस लागू करता है। सॉफ़्टवेयर को कंटेनराइज़्ड डिप्लॉयमेंट के लिए होस्ट नेटवर्किंग और हार्डवेयर डिवाइस मैपिंग के सपोर्ट के साथ पैक किया गया है।

KServe is an open platform for deploying and serving generative and predictive AI models on Kubernetes. It defines inference services as custom resources with declarative YAML specifications, enabling a Kubernetes-native approach to model deployment and lifecycle management. The platform leverages Knative-based serverless scaling for automatic scale-to-zero and revision management, and supports a pluggable serving runtime architecture that maps model formats to containerized execution environments. KServe distinguishes itself through model-aware autoscaling that scales replicas based on token

KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere

SLIME is a distributed reinforcement learning framework for large language model post-training that bridges Megatron training with SGLang inference servers. It orchestrates scalable RL loops across GPU clusters, decoupling training and inference into independent processes that communicate over HTTP and NCCL for independent scaling and fault tolerance. The system supports multi-agent reinforcement learning workflows with parallel agent instances, customizable rollout strategies, and personalized agent serving that improves models from prior conversations without disrupting API serving. The fra

LightLLM is a high-performance serving framework for deploying and executing large language models. It functions as a multi-GPU inference engine and server capable of handling dense architectures, mixture-of-experts designs, and multimodal models that process both text and images. The system is distinguished by its specialized support for Mixture-of-Experts models using expert parallelism and fused kernels. It implements structured text generation through deterministic state machines and pushdown automata to enforce precise output formats. To optimize throughput, the framework employs specula

FastDeploy is a high-performance deployment framework for large language models, vision models, and multimodal models. It provides the infrastructure to launch model services that process combined image, video, and text inputs, exposing these capabilities through a standardized, OpenAI-compatible API for chat and text completions. The project distinguishes itself through advanced inference pipeline engineering and GPU optimization. It employs speculative decoding, tensor parallelism, and a disaggregated execution model that separates prefill and decode phases across different hardware resourc