2 Repos
Awesome GitHub RepositoriesLLM Inference Orchestration
Orchestrating large language model inference tasks on GPU hardware within a Kubernetes cluster.
Distinct from GPU Workload Virtualization and Containerization: Distinct from general GPU Workload Virtualization and Containerization: focuses specifically on orchestrating LLM inference tasks, not general GPU workload management.
Explore 2 awesome GitHub repositories matching operating systems & systems programming · LLM Inference Orchestration. Refine with filters or upvote what's useful.
Awesome LLM Inference Orchestration GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- wasmedge/wasmedge
WasmEdge/WasmEdge
10,665Auf GitHub ansehenWasmEdge is an extensible WebAssembly runtime that executes WebAssembly bytecode in a secure sandbox for cloud, edge, and embedded applications. It functions as a multi-language compiler, compiling applications written in Rust, JavaScript, Go, and Python into WebAssembly bytecode for sandboxed execution, and as a server-side JavaScript runtime that runs JavaScript programs with ES6 modules, NPM packages, and Node.js-compatible APIs. The runtime also serves as an AI inference runtime, executing AI models from JavaScript using WASI-NN plug-ins for inference tasks on personal devices and edge har
Orchestrates large language model inference tasks on GPU hardware within a Kubernetes cluster.
C++artificial-intelligencecloudcloud-native - gpustack/gpustack
gpustack/gpustack
5,173Auf GitHub ansehengpustack is a GPU cluster management platform and LLM inference orchestrator. It functions as a centralized system for pooling and orchestrating graphics processing units across local servers and cloud environments, serving as a heterogeneous compute manager for diverse hardware and software configurations. The system provides a secure AI model deployment gateway that serves models as scalable services using key-based authentication. It includes a GPU resource scheduler that balances workloads across accelerators and coordinates multiple inference engines to map specific AI models to compatib
Coordinates and deploys inference engines like vLLM and SGLang to serve AI models.
Python
