1 مستودع
Awesome GitHub RepositoriesProcess Isolations
Mechanisms to isolate the cache management process from the inference engine to prevent data loss during crashes.
Distinct from Inference Cache Management: Focuses on process-level architectural isolation rather than memory buffer allocation for inference.
Explore 1 awesome GitHub repository matching operating systems & systems programming · Process Isolations. Refine with filters or upvote what's useful.
Awesome Process Isolations GitHub Repositories
اعثر على أفضل المستودعات باستخدام الذكاء الاصطناعي.سنبحث عن أفضل المستودعات المطابقة باستخدام الذكاء الاصطناعي.
- lmcache/lmcache
LMCache/LMCache
6,909عرض على GitHubLMCache is a distributed key-value cache manager and tiering system designed to accelerate large language model inference. It functions as a tiered storage layer that offloads tensors from GPU memory to CPU RAM, local disks, or remote object stores, enabling the reuse of cached prefixes across different inference sessions and serving engines. The system differentiates itself through a disaggregated prefill-decode model, which separates prompt processing from token generation by transferring caches between distributed compute nodes. It utilizes peer-to-peer orchestration to share and retrieve
Runs the cache manager as a standalone daemon to ensure cache persistence during inference engine crashes.
Pythonamdcudafast
