1 repository
Awesome GitHub RepositoriesRequest Migrations
The process of moving active processing tasks from one worker to another without losing state.
Distinct from Request Cancellation: Focuses on migrating active requests between healthy nodes rather than simply cancelling them.
Explore 1 awesome GitHub repository matching web development · Request Migrations. Refine with filters or upvote what's useful.
Awesome Request Migrations GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- ai-dynamo/dynamo
ai-dynamo/dynamo
6,112Vezi pe GitHubDynamo is a distributed inference orchestration platform designed for large language models. It functions as a system to coordinate prefill and decode phases across GPU nodes, utilizing a multi-backend runtime adapter to connect engines like vLLM and TensorRT-LLM through a unified block-oriented memory interface. An OpenAI-compatible API server provides the frontend for integration with existing tools and clients. The project is distinguished by its disaggregated serving architecture, which separates prompt processing and token generation onto independent GPU pools to optimize throughput and
Moves active requests to healthy workers during failures to prevent request loss.
Rust
