higress-group/higress

Features

• AI Gateways - Provides a unified AI-native gateway for routing, securing, and managing traffic to multiple LLM providers.
• AI Service Gateways - Provides a cloud-native gateway that routes, secures, and optimizes traffic specifically for large language model services.
• Traffic Management - Balances requests across multiple AI providers using unified protocols, token-aware rate limiting, and response caching.
• AI Traffic Routing - Provides specialized routing and management for traffic directed toward machine learning applications.
• Ingress Controllers - Implements standardized ingress APIs to manage external access and network routing within clusters.
• Token-Based Rate Limiters - Enforces rate limits based on token consumption per client to prevent service exhaustion in LLM endpoints.
• AI Model Load Balancers - Balances requests across multiple AI model providers based on token costs, latency, and availability.
• Centralized API Gateways - Provides a centralized entry point to route incoming network requests to the correct backend microservices.
• AI Protocol Translation - Translates diverse AI provider APIs into a single standardized protocol for seamless routing.
• AI Model Routing Strategies - Employs logic to distribute LLM requests across various providers to optimize for cost and speed.
• API Gateways - Serves as a centralized entry point for microservices with integrated service discovery and request throttling.
• Model Context Protocol Integrations - Implements Model Context Protocol integrations to expose real-time system data and functions to AI models.
• Inference Response Optimizations - Improves the delivery of AI responses by directing traffic to efficient endpoints and caching results.
• Inference Traffic Optimizations - Directs traffic to efficient inference endpoints to ensure the fast delivery of AI-generated responses.
• Memory-Efficient Data Streaming - Processes request and response bodies as continuous data streams to minimize memory overhead for AI responses.
• Traffic Transformation Modules - Deploys lightweight binary modules to intercept and transform network traffic at the routing level.
• Request Processing Plugins - Provides modular plugins that process and transform incoming HTTP requests using custom logic.
• Sandboxed Plugin Execution - Executes request-handling plugins in a sandboxed environment with hot-reloading to ensure system stability and security.
• MCP Server Hosting - Provides specialized infrastructure for hosting Model Context Protocol servers to make tools available to AI agents.
• Microservice Traffic Management - Routes requests to distributed microservices by discovering service locations from external registries.
• Service Registry Integrations - Integrates with service registries to automatically locate and route requests to available backend instances.
• Service Registries - Integrates with service registries to dynamically discover and route traffic to backend microservices.
• Traffic Throttling - Limits the volume of incoming traffic using specific parameters to prevent service exhaustion under high load.
• Per-Client Request Throttlers - Protects backend services from abuse by restricting the number of requests allowed per client identity.
• Chunked Response Processing - Processes request and response bodies as continuous data flows to minimize memory overhead for high-bandwidth AI traffic.
• Traffic Routing Controllers - Manages the flow and security of requests to language model services using a specialized control plane.
• API Access Security - Secures API endpoints by combining firewall rules with multiple identity verification strategies.
• TLS Certificate Management - Automates the issuance and renewal of TLS certificates to ensure secure encrypted connections.
• API Request Authentication - Validates request credentials using standard authentication schemes to protect sensitive resources.
• Automated Certificate Management - Automates the issuance and renewal of TLS certificates for secure, encrypted connections.
• Traffic Security Policies - Enforces security policies and access controls on traffic flows to keep sensitive services protected.
• Traffic Filtering - Filters common web attacks through a security firewall to protect backend microservices from exploitation.
• Web Application Firewalls - Implements a web application firewall to filter and block malicious HTTP traffic protecting backend services.
• Wasm-Based Plugins - Executes custom logic in a sandboxed WebAssembly runtime to transform network traffic with hot-reloading.
• API Gateways - Cloud-native gateway based on Envoy and Istio.

Star history