Awesome GitHub RepositoriesInference Gateways

Odysseus is a self-hosted AI workspace and autonomous agent framework designed for deploying and managing large language models. It serves as a centralized platform for orchestrating agentic tasks, utilizing a model context protocol server to connect AI models to external system utilities, browser automation, and local hardware. The system distinguishes itself through a combination of retrieval-augmented generation and a RAG knowledge base, using vector stores and local embeddings to provide persistent semantic memory. It further integrates AI-driven communication management to triage email i

This project is a comprehensive infrastructure guide and technical reference for designing and deploying cloud native and AI native environments using Kubernetes. It serves as a manual for managing container orchestration, pod lifecycles, and declarative state reconciliation to maintain scalable cluster workloads. The resource provides instructional material on building custom controllers and implementing operational logic via the operator pattern. It also functions as a framework for optimizing the delivery of large language models through specialized gateways and workload scheduling. The h

Higress 是一个 AI 原生和云原生的 API 网关，用于路由、保护和优化客户端与大语言模型服务之间的流量。它作为微服务的集中入口点，同时充当 Kubernetes Ingress 控制器和 AI 网关编排器。 该项目通过使用统一协议管理跨多个 AI 提供商的流量而脱颖而出，结合了令牌感知速率限制和响应缓存以优化模型推理。它协调 AI 模型与外部工具之间的通信以提供实时上下文和数据，同时还为 AI 代理托管服务器端点。 广泛的功能包括通过 Web 应用防火墙（WAF）实施 API 安全、自动 TLS 证书管理和动态服务发现。该网关通过沙箱化的 WebAssembly 插件支持自定义请求处理，允许通过热重载进行流量转换。 该系统实现了标准化的 Ingress API，以低资源开销管理容器化集群内的网络路由。

Spin is a WebAssembly serverless framework and development toolchain for building and running portable microservices. It functions as an event-driven orchestrator and runtime that executes WebAssembly components, allowing developers to map HTTP requests, Redis messages, and cron schedules to specific modules. The project distinguishes itself by implementing a Wasm-based AI inference gateway, enabling components to perform model inference and generate text embeddings. It utilizes the WebAssembly Component Model and WASI for language-agnostic composition and portable host interfacing, while emp

Helicone is an AI gateway and observability platform designed to intercept, manage, and monitor interactions with large language models. By acting as a reverse-proxy, it provides a centralized layer for routing requests across multiple AI providers, allowing developers to maintain consistent application logic while gaining deep visibility into model performance, usage, and costs. The platform distinguishes itself through a robust suite of traffic management and prompt engineering tools. It enables policy-driven control, including automatic failover between providers, rate limiting, and edge-b

KServe is an open platform for deploying and serving generative and predictive AI models on Kubernetes. It defines inference services as custom resources with declarative YAML specifications, enabling a Kubernetes-native approach to model deployment and lifecycle management. The platform leverages Knative-based serverless scaling for automatic scale-to-zero and revision management, and supports a pluggable serving runtime architecture that maps model formats to containerized execution environments. KServe distinguishes itself through model-aware autoscaling that scales replicas based on token

KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere

Aibrix 是一个推理编排器，专为跨分布式 vLLM 集群扩展、路由和管理大语言模型部署而设计。它作为一个集中式网关，用于负载均衡并将流量路由到特定的模型副本和版本。 该系统通过 GPU 集群自动缩放器管理资源效率，该缩放器根据实时请求量调整计算实例数量。它通过在单个集群内混合不同加速器类型，并利用模型适配器编排器在共享基础模型上部署轻量级参数适配器，进一步优化了操作。 广泛的功能包括使用分布式键值缓存管理器在推理引擎之间共享 Token 数据，以及实施硬件健康监控以检测处理单元故障。该项目还提供了一个统一的指标流水线，以标准化跨不同运行时环境的性能数据收集。

Seldon Core 是一个基于 Kubernetes 的机器学习模型服务器和 MLOps 推理框架。它作为多模型服务引擎和流水线编排器，将模型打包为可通过标准化 REST 和 gRPC API 暴露的可扩展微服务。 该项目的独特之处在于基于图的推理流水线，可将模型和数据转换器链接成顺序工作流。它通过多模型共享服务和动态内存超额分配策略优化硬件利用率，同时通过加权流量路由、A/B 测试和影子部署支持生产环境实验。 该框架涵盖了广泛的 MLOps 功能，包括基于需求的自动扩缩容、通过消息总线进行的异步请求处理，以及针对数据漂移、异常值和预测可解释性的全面监控。它还提供用于模型运行时配置的基础设施管理，以及跨控制平面和数据平面的 TLS 加密安全通信。

llm-d is a distributed serving framework designed for large language model inference. It functions as an inference orchestrator and gateway, providing a control plane for deploying model replicas and managing hardware accelerators. The system includes a batch inference scheduler and a cache manager to coordinate request flow and memory utilization. The project is distinguished by a disaggregated serving architecture that separates prefill and decode execution phases across specialized workers to maximize throughput. It employs a hardware-agnostic control plane and tiered cache offloading, mov