Awesome GitHub RepositoriesInference Gateways

Odysseus is a self-hosted AI workspace and autonomous agent framework designed for deploying and managing large language models. It serves as a centralized platform for orchestrating agentic tasks, utilizing a model context protocol server to connect AI models to external system utilities, browser automation, and local hardware. The system distinguishes itself through a combination of retrieval-augmented generation and a RAG knowledge base, using vector stores and local embeddings to provide persistent semantic memory. It further integrates AI-driven communication management to triage email i

This project is a comprehensive infrastructure guide and technical reference for designing and deploying cloud native and AI native environments using Kubernetes. It serves as a manual for managing container orchestration, pod lifecycles, and declarative state reconciliation to maintain scalable cluster workloads. The resource provides instructional material on building custom controllers and implementing operational logic via the operator pattern. It also functions as a framework for optimizing the delivery of large language models through specialized gateways and workload scheduling. The h

Higress este un API gateway AI-native și cloud-native care rutează, securizează și optimizează traficul între clienți și serviciile de modele de limbaj mari (LLM). Funcționează ca un punct de intrare centralizat pentru microservicii, servind atât ca un controller de ingress Kubernetes, cât și ca un orchestrator de gateway AI. Proiectul se distinge prin gestionarea traficului între mai mulți furnizori AI folosind un protocol unificat, încorporând limitarea ratei de acces (rate limiting) conștientă de token-uri și caching-ul răspunsurilor pentru a optimiza inferența modelului. Coordonează comunicarea între modelele AI și instrumentele externe pentru a oferi context și date în timp real, găzduind în același timp endpoint-uri de server pentru agenți AI. Capabilitățile largi includ aplicarea securității API prin firewall-uri pentru aplicații web (WAF), gestionarea automatizată a certificatelor TLS și descoperirea dinamică a serviciilor. Gateway-ul suportă procesarea personalizată a cererilor prin plugin-uri WebAssembly sandbox-uite care permit transformarea traficului cu hot-reloading. Sistemul implementează API-uri de ingress standardizate pentru a gestiona rutarea rețelei în clustere containerizate cu un overhead de resurse redus.

Spin is a WebAssembly serverless framework and development toolchain for building and running portable microservices. It functions as an event-driven orchestrator and runtime that executes WebAssembly components, allowing developers to map HTTP requests, Redis messages, and cron schedules to specific modules. The project distinguishes itself by implementing a Wasm-based AI inference gateway, enabling components to perform model inference and generate text embeddings. It utilizes the WebAssembly Component Model and WASI for language-agnostic composition and portable host interfacing, while emp

Helicone is an AI gateway and observability platform designed to intercept, manage, and monitor interactions with large language models. By acting as a reverse-proxy, it provides a centralized layer for routing requests across multiple AI providers, allowing developers to maintain consistent application logic while gaining deep visibility into model performance, usage, and costs. The platform distinguishes itself through a robust suite of traffic management and prompt engineering tools. It enables policy-driven control, including automatic failover between providers, rate limiting, and edge-b

KServe is an open platform for deploying and serving generative and predictive AI models on Kubernetes. It defines inference services as custom resources with declarative YAML specifications, enabling a Kubernetes-native approach to model deployment and lifecycle management. The platform leverages Knative-based serverless scaling for automatic scale-to-zero and revision management, and supports a pluggable serving runtime architecture that maps model formats to containerized execution environments. KServe distinguishes itself through model-aware autoscaling that scales replicas based on token

KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere

Aibrix is an inference orchestrator designed for scaling, routing, and managing the deployment of large language models across distributed vLLM clusters. It serves as a centralized gateway for load balancing and routing traffic to specific model replicas and versions. The system manages resource efficiency through a GPU cluster autoscaler that adjusts compute instance counts based on real-time request volume. It further optimizes operations by mixing different accelerator types within a single cluster and utilizing a model adapter orchestrator to deploy lightweight parameter adapters on share

Seldon Core este un server de modele de machine learning bazat pe Kubernetes și un framework de inferență MLOps. Funcționează ca un motor de servire multi-model și orchestrator de pipeline-uri, împachetând modelele ca microservicii scalabile care sunt expuse prin API-uri standardizate REST și gRPC. Proiectul se distinge prin pipeline-uri de inferență bazate pe grafuri care înlănțuie modele și transformatoare de date în fluxuri de lucru secvențiale. Optimizează utilizarea hardware-ului prin servire partajată multi-model și strategii de overcommit dinamic al memoriei, susținând în același timp experimentarea în producție prin rutarea ponderată a traficului, testarea A/B și deployment-uri de tip shadow. Framework-ul acoperă o gamă largă de capabilități MLOps, inclusiv autoscaling bazat pe cerere, procesarea asincronă a cererilor prin message bus-uri și monitorizarea completă pentru data drift, valori aberante (outliers) și explicabilitatea predicțiilor. Oferă, de asemenea, gestionarea infrastructurii pentru configurarea runtime-ului modelelor și comunicare securizată folosind criptare TLS pe planurile de control și de date.

llm-d is a distributed serving framework designed for large language model inference. It functions as an inference orchestrator and gateway, providing a control plane for deploying model replicas and managing hardware accelerators. The system includes a batch inference scheduler and a cache manager to coordinate request flow and memory utilization. The project is distinguished by a disaggregated serving architecture that separates prefill and decode execution phases across specialized workers to maximize throughput. It employs a hardware-agnostic control plane and tiered cache offloading, mov