3 repository-uri
Awesome GitHub RepositoriesWorkload Simulations
Tools for simulating specific traffic patterns to measure inference throughput and speed.
Distinct from Inference Speed Profiling: Focuses on synthetic traffic generation and load testing rather than just timing profiling.
Explore 3 awesome GitHub repositories matching artificial intelligence & ml · Workload Simulations. Refine with filters or upvote what's useful.
Awesome Workload Simulations GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- lmcache/lmcache
LMCache/LMCache
6,909Vezi pe GitHubLMCache is a distributed key-value cache manager and tiering system designed to accelerate large language model inference. It functions as a tiered storage layer that offloads tensors from GPU memory to CPU RAM, local disks, or remote object stores, enabling the reuse of cached prefixes across different inference sessions and serving engines. The system differentiates itself through a disaggregated prefill-decode model, which separates prompt processing from token generation by transferring caches between distributed compute nodes. It utilizes peer-to-peer orchestration to share and retrieve
Simulates configurable traffic patterns to report speed and throughput metrics for the inference engine.
Pythonamdcudafast - ai-dynamo/dynamo
ai-dynamo/dynamo
6,112Vezi pe GitHubDynamo is a distributed inference orchestration platform designed for large language models. It functions as a system to coordinate prefill and decode phases across GPU nodes, utilizing a multi-backend runtime adapter to connect engines like vLLM and TensorRT-LLM through a unified block-oriented memory interface. An OpenAI-compatible API server provides the frontend for integration with existing tools and clients. The project is distinguished by its disaggregated serving architecture, which separates prompt processing and token generation onto independent GPU pools to optimize throughput and
Mimics backend API behavior and synthetic traffic patterns to validate routing and infrastructure logic without consuming GPUs.
Rust - alexcasalboni/aws-lambda-power-tuning
alexcasalboni/aws-lambda-power-tuning
6,028Vezi pe GitHubThis project is a performance optimizer and resource benchmarker for AWS Lambda. It analyzes the trade-off between execution speed and cost by testing various memory configurations to identify the most cost-effective settings and minimize operational spending. The tool utilizes an AWS Step Functions orchestrator to automate the execution and data collection of multiple function test runs across different power levels. It simulates production workloads by injecting custom static or remote data and using weighted payload distribution to mimic real-world traffic patterns. The suite covers sever
Simulates production traffic by distributing test input payloads based on assigned relative probability weights.
JavaScript
