1 Repo
Awesome GitHub RepositoriesKV-Cache Transport Optimizations
Communication systems and protocols designed to accelerate the movement of KV cache blocks between nodes.
Distinct from Activation and KV Cache Offloaders: Focuses on the transport/migration of cache rather than storage, offloading, or quantization
Explore 1 awesome GitHub repository matching artificial intelligence & ml · KV-Cache Transport Optimizations. Refine with filters or upvote what's useful.
Awesome KV-Cache Transport Optimizations GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- llm-d/llm-d
llm-d/llm-d
2,514Auf GitHub ansehenllm-d is a distributed serving framework designed for large language model inference. It functions as an inference orchestrator and gateway, providing a control plane for deploying model replicas and managing hardware accelerators. The system includes a batch inference scheduler and a cache manager to coordinate request flow and memory utilization. The project is distinguished by a disaggregated serving architecture that separates prefill and decode execution phases across specialized workers to maximize throughput. It employs a hardware-agnostic control plane and tiered cache offloading, mov
Reduces tail latency during context migration using adaptive congestion control and specialized libraries.
Shell
