Awesome GitHub RepositoriesInference Gateways

Odysseus is a self-hosted AI workspace and autonomous agent framework designed for deploying and managing large language models. It serves as a centralized platform for orchestrating agentic tasks, utilizing a model context protocol server to connect AI models to external system utilities, browser automation, and local hardware. The system distinguishes itself through a combination of retrieval-augmented generation and a RAG knowledge base, using vector stores and local embeddings to provide persistent semantic memory. It further integrates AI-driven communication management to triage email i

This project is a comprehensive infrastructure guide and technical reference for designing and deploying cloud native and AI native environments using Kubernetes. It serves as a manual for managing container orchestration, pod lifecycles, and declarative state reconciliation to maintain scalable cluster workloads. The resource provides instructional material on building custom controllers and implementing operational logic via the operator pattern. It also functions as a framework for optimizing the delivery of large language models through specialized gateways and workload scheduling. The h

Higress is an AI-native and cloud-native API gateway that routes, secures, and optimizes traffic between clients and large language model services. It functions as a centralized entry point for microservices, serving as both a Kubernetes ingress controller and an AI gateway orchestrator. The project distinguishes itself by managing traffic across multiple AI providers using a unified protocol, incorporating token-aware rate limiting and response caching to optimize model inference. It coordinates communication between AI models and external tools to provide real-time context and data, while a

Spin is a WebAssembly serverless framework and development toolchain for building and running portable microservices. It functions as an event-driven orchestrator and runtime that executes WebAssembly components, allowing developers to map HTTP requests, Redis messages, and cron schedules to specific modules. The project distinguishes itself by implementing a Wasm-based AI inference gateway, enabling components to perform model inference and generate text embeddings. It utilizes the WebAssembly Component Model and WASI for language-agnostic composition and portable host interfacing, while emp

Helicone is an AI gateway and observability platform designed to intercept, manage, and monitor interactions with large language models. By acting as a reverse-proxy, it provides a centralized layer for routing requests across multiple AI providers, allowing developers to maintain consistent application logic while gaining deep visibility into model performance, usage, and costs. The platform distinguishes itself through a robust suite of traffic management and prompt engineering tools. It enables policy-driven control, including automatic failover between providers, rate limiting, and edge-b

KServe is an open platform for deploying and serving generative and predictive AI models on Kubernetes. It defines inference services as custom resources with declarative YAML specifications, enabling a Kubernetes-native approach to model deployment and lifecycle management. The platform leverages Knative-based serverless scaling for automatic scale-to-zero and revision management, and supports a pluggable serving runtime architecture that maps model formats to containerized execution environments. KServe distinguishes itself through model-aware autoscaling that scales replicas based on token

KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere

Aibrix is an inference orchestrator designed for scaling, routing, and managing the deployment of large language models across distributed vLLM clusters. It serves as a centralized gateway for load balancing and routing traffic to specific model replicas and versions. The system manages resource efficiency through a GPU cluster autoscaler that adjusts compute instance counts based on real-time request volume. It further optimizes operations by mixing different accelerator types within a single cluster and utilizing a model adapter orchestrator to deploy lightweight parameter adapters on share

Seldon Core هو خادم نماذج تعلم آلي وإطار عمل للاستدلال (Inference) يعتمد على Kubernetes. يعمل كمحرك لخدمة النماذج المتعددة ومنسق لخطوط الأنابيب، حيث يقوم بتغليف النماذج كخدمات مصغرة قابلة للتوسع يتم عرضها عبر واجهات REST وgRPC قياسية. يتميز المشروع بخطوط أنابيب استدلال قائمة على الرسوم البيانية (Graph-based) تقوم بربط النماذج ومحولات البيانات في سير عمل متسلسل. يعمل على تحسين استخدام الأجهزة من خلال الخدمة المشتركة للنماذج المتعددة واستراتيجيات تجاوز سعة الذاكرة الديناميكية، مع دعم تجارب الإنتاج من خلال توجيه حركة المرور الموزون، واختبار A/B، والنشر الظلي (Shadow deployments). يغطي إطار العمل نطاقاً واسعاً من قدرات MLOps، بما في ذلك القياس التلقائي القائم على الطلب، ومعالجة الطلبات غير المتزامنة عبر ناقلات الرسائل، والمراقبة الشاملة لانحراف البيانات، والقيم المتطرفة، وقابلية تفسير التنبؤات. كما يوفر إدارة البنية التحتية لتكوين وقت تشغيل النموذج والاتصال الآمن باستخدام تشفير TLS عبر مستويات التحكم والبيانات.

llm-d is a distributed serving framework designed for large language model inference. It functions as an inference orchestrator and gateway, providing a control plane for deploying model replicas and managing hardware accelerators. The system includes a batch inference scheduler and a cache manager to coordinate request flow and memory utilization. The project is distinguished by a disaggregated serving architecture that separates prefill and decode execution phases across specialized workers to maximize throughput. It employs a hardware-agnostic control plane and tiered cache offloading, mov