Awesome GitHub RepositoriesInference Cache Management

Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr

ChatGLM2-6B is an open-weight large language model designed for natural language conversations and text generation in both English and Chinese. It functions as a bilingual chat model capable of processing and maintaining coherence across text sequences up to 32K tokens. The model is optimized for local deployment through precision quantization, which reduces memory requirements to allow execution on consumer-grade hardware. It supports distributing model weights across multiple graphics cards to handle parameters that exceed the memory of a single device. The project covers capabilities for

RWKV-LM is a framework for training and deploying recurrent language models. It utilizes a linear-time recurrent architecture that enables text generation and sequence processing with constant memory and time complexity, avoiding the quadratic scaling of traditional attention caches. The project implements a parallelizable training mechanism that allows recurrent models to be trained using global operations while maintaining cache-free inference. It includes state-tuning capabilities to optimize the initial hidden state and utilizes adaptive probability-mass sampling to control token diversit

This project serves as a comprehensive educational resource and technical handbook for engineers building applications powered by large language models. It provides a structured framework for mastering the principles of artificial intelligence engineering, covering the full lifecycle of model development from initial design to production deployment. The repository distinguishes itself by offering a deep dive into the practical implementation of advanced design patterns, including retrieval-augmented generation, agentic tool orchestration, and parameter-efficient model adaptation. It emphasize

ChatGLM3 is a comprehensive framework for deploying, fine-tuning, and serving large language models. It functions as a high-performance inference engine designed to support conversational AI, enabling developers to build interactive agents capable of multi-turn dialogue, autonomous code execution, and structured tool invocation. The project distinguishes itself through its focus on hardware-agnostic deployment and resource optimization. It supports distributed model parallelism across multiple graphics cards, paged key-value caching for concurrent request processing, and weight quantization t

Lingbot-map is a feed-forward neural network designed for real-time 3D scene reconstruction from streaming video. It processes video frames one at a time without iterative optimization, producing dense geometry and camera poses at interactive frame rates directly from a live feed. The project distinguishes itself through its ability to maintain stable geometry and pose alignment across very long video sequences, handling thousands of frames without drift. It achieves this through a combination of coordinate grounding memory, sliding-window inference with overlapping keyframes, and a paged KV

LMCache is a distributed key-value cache manager and tiering system designed to accelerate large language model inference. It functions as a tiered storage layer that offloads tensors from GPU memory to CPU RAM, local disks, or remote object stores, enabling the reuse of cached prefixes across different inference sessions and serving engines. The system differentiates itself through a disaggregated prefill-decode model, which separates prompt processing from token generation by transferring caches between distributed compute nodes. It utilizes peer-to-peer orchestration to share and retrieve

Mooncake ist eine disaggregierte Plattform für das Serving von Large Language Models und ein verteilter Key-Value-Store, der für eine hochperformante Inferenz-Infrastruktur konzipiert wurde. Es fungiert als GPU-Speicher-Orchestrator und KV-Cache-Managementsystem, das Key-Value-Caches über Cluster hinweg bündelt und überträgt, um die Inferenz zu beschleunigen. Das System zeichnet sich dadurch aus, dass es die Prefill- und Decode-Phasen der Inferenz in separate Hardware-Cluster trennt, um die Ressourcennutzung zu optimieren. Es nutzt einen hochperformanten verteilten RDMA-Cache mit Zero-Copy-Transfers, um Daten zwischen Rechenknoten zu verschieben und dabei die CPU zu umgehen, um Latenz und Overhead zu reduzieren. Die Plattform deckt breite Funktionsbereiche ab, einschließlich verteiltem Memory-Pooling, Beschleuniger-Speicher-Routing via CXL und Multi-Tier-Storage-Offloading auf SSDs. Es verwaltet den Cluster-Status durch Metadaten-Koordinationsdienste und implementiert Ressourcen-Governance mittels lease-basierter Objektschutzmechanismen und wasserzeichenbasierter Cache-Eviction. Die Software ist für containerisierte Deployments verpackt, mit Unterstützung für Host-Networking und Hardware-Device-Mapping.

This project provides a comprehensive technical guide and framework for engineering large-scale machine learning systems. It covers the full lifecycle of model development, focusing on the infrastructure and computational principles required to build, train, and serve generative AI models across distributed GPU clusters. The repository distinguishes itself by offering deep-dive tutorials and implementation strategies for complex system challenges. It emphasizes high-performance architectural primitives, such as collective communication orchestration, distributed tensor sharding, and static gr

Dieses Projekt ist ein ausdrucksstarkes Text-to-Speech-Grundlagenmodell und Voice-Cloning-System, das darauf ausgelegt ist, menschenähnliche Sprache mit emotionaler Nuance und hoher Wiedergabetreue zu synthetisieren. Es fungiert als feinabstimmbares Sprachmodell, das Audio generieren kann, das eine bestimmte Person unter Verwendung eines Referenz-Stimmbeispiels imitiert. Das System zeichnet sich durch eine hochperformante Inference-Engine aus, die Memory-Caching und Hardware-Kompilierung nutzt, um die Latenz während des Audio-Generierungsprozesses zu reduzieren. Es ermöglicht zudem Verbesserungen der Synthesequalität durch das Training des Sprachmodells auf benutzerdefinierten Datensätzen, die aus Audiodateien und passenden Untertiteln bestehen. Das Framework deckt die breiteren Bereiche des benutzerdefinierten Voice-Clonings, der ausdrucksstarken Sprachsynthese und der Feinabstimmung von Sprachmodellen ab.