Awesome GitHub RepositoriesInference Cache Management

Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr

ChatGLM2-6B is an open-weight large language model designed for natural language conversations and text generation in both English and Chinese. It functions as a bilingual chat model capable of processing and maintaining coherence across text sequences up to 32K tokens. The model is optimized for local deployment through precision quantization, which reduces memory requirements to allow execution on consumer-grade hardware. It supports distributing model weights across multiple graphics cards to handle parameters that exceed the memory of a single device. The project covers capabilities for

RWKV-LM is a framework for training and deploying recurrent language models. It utilizes a linear-time recurrent architecture that enables text generation and sequence processing with constant memory and time complexity, avoiding the quadratic scaling of traditional attention caches. The project implements a parallelizable training mechanism that allows recurrent models to be trained using global operations while maintaining cache-free inference. It includes state-tuning capabilities to optimize the initial hidden state and utilizes adaptive probability-mass sampling to control token diversit

This project serves as a comprehensive educational resource and technical handbook for engineers building applications powered by large language models. It provides a structured framework for mastering the principles of artificial intelligence engineering, covering the full lifecycle of model development from initial design to production deployment. The repository distinguishes itself by offering a deep dive into the practical implementation of advanced design patterns, including retrieval-augmented generation, agentic tool orchestration, and parameter-efficient model adaptation. It emphasize

ChatGLM3 is a comprehensive framework for deploying, fine-tuning, and serving large language models. It functions as a high-performance inference engine designed to support conversational AI, enabling developers to build interactive agents capable of multi-turn dialogue, autonomous code execution, and structured tool invocation. The project distinguishes itself through its focus on hardware-agnostic deployment and resource optimization. It supports distributed model parallelism across multiple graphics cards, paged key-value caching for concurrent request processing, and weight quantization t

Lingbot-map is a feed-forward neural network designed for real-time 3D scene reconstruction from streaming video. It processes video frames one at a time without iterative optimization, producing dense geometry and camera poses at interactive frame rates directly from a live feed. The project distinguishes itself through its ability to maintain stable geometry and pose alignment across very long video sequences, handling thousands of frames without drift. It achieves this through a combination of coordinate grounding memory, sliding-window inference with overlapping keyframes, and a paged KV

LMCache is a distributed key-value cache manager and tiering system designed to accelerate large language model inference. It functions as a tiered storage layer that offloads tensors from GPU memory to CPU RAM, local disks, or remote object stores, enabling the reuse of cached prefixes across different inference sessions and serving engines. The system differentiates itself through a disaggregated prefill-decode model, which separates prompt processing from token generation by transferring caches between distributed compute nodes. It utilizes peer-to-peer orchestration to share and retrieve

Mooncake este o platformă de servire a modelelor de limbaj mari (LLM) dezagregate și un magazin distribuit de tip cheie-valoare, conceput pentru infrastructura de inferență de înaltă performanță. Funcționează ca un orchestrator de memorie GPU și sistem de gestionare a cache-ului KV care grupează și transferă cache-urile cheie-valoare între clustere pentru a accelera inferența. Sistemul se distinge prin separarea fazelor de prefill și decode ale inferenței în clustere hardware distincte pentru a optimiza utilizarea resurselor. Utilizează un cache distribuit RDMA de înaltă performanță cu transferuri zero-copy pentru a muta datele între nodurile de calcul, ocolind CPU-ul pentru a reduce latența și overhead-ul. Platforma acoperă domenii largi de capabilități, inclusiv gruparea memoriei distribuite, rutarea memoriei acceleratorului prin CXL și descărcarea stocării pe mai multe niveluri către SSD-uri. Gestionează starea clusterului prin servicii de coordonare a metadatelor și implementează guvernanța resurselor prin protecția obiectelor bazată pe lease și evacuarea cache-ului bazată pe watermark. Software-ul este împachetat pentru deployment containerizat cu suport pentru rețelistică host și maparea dispozitivelor hardware.

This project provides a comprehensive technical guide and framework for engineering large-scale machine learning systems. It covers the full lifecycle of model development, focusing on the infrastructure and computational principles required to build, train, and serve generative AI models across distributed GPU clusters. The repository distinguishes itself by offering deep-dive tutorials and implementation strategies for complex system challenges. It emphasizes high-performance architectural primitives, such as collective communication orchestration, distributed tensor sharding, and static gr

Acest proiect este un model de bază expresiv de text-to-speech și un sistem de clonare a vocii, conceput pentru a sintetiza vorbirea umană cu nuanțe emoționale și înaltă fidelitate. Acesta funcționează ca un model de vorbire finetunabil care poate genera audio imitând o anumită persoană folosind o mostră de voce de referință. Sistemul se distinge printr-un motor de inferență de înaltă performanță care utilizează caching-ul memoriei și compilarea hardware pentru a reduce latența în timpul procesului de generare audio. De asemenea, permite îmbunătățirea calității sintezei prin antrenarea modelului de limbaj pe seturi de date personalizate constând în fișiere audio și subtitrări corespondente. Framework-ul acoperă domeniile mai largi ale clonării vocale personalizate, sintezei vocale expresive și finetuning-ului modelelor de vorbire.