alibaba/higress

Features

• AI API Gateways - Provides a specialized gateway for routing, securing, and optimizing traffic specifically for large language models and AI agents.
• AI Gateway Management - Provides a centralized AI gateway for routing, load balancing, and token quota management for LLMs.
• Traffic Management - Provides a specialized gateway for proxying, auditing, and securing traffic specifically for AI models.
• Ingress Controllers - Functions as a Kubernetes ingress controller supporting the Gateway API and Nginx Ingress annotations.
• Model Context Protocol - Converts requests from a single interface into the specific protocols required by various AI models.
• MCP Server Management - Provides interfaces for registering and organizing MCP server connections from registries and external providers.
• AI Agent Orchestration - Implements orchestration templates to deploy customizable API agents that execute external tools via web requests.
• AI Tool Converters - Transforms existing REST APIs into callable AI tools by mapping requests to HTTP calls using templates.
• Retrieval-Augmented Generation - Augments model responses by retrieving relevant documents from vector search services to provide factual context.
• Token Quota Enforcement - Implements global and dynamic token consumption limits to manage AI model costs and resource usage.
• Semantic Caching Systems - Stores and retrieves model responses using vector-based semantic matching to reduce token consumption and latency.
• MCP Servers - Provides implementations of the Model Context Protocol to expose databases and APIs as tools for AI agents.
• Model Context Protocol Servers - Implements standardized interfaces to connect AI models to external tools and data sources via MCP.
• Model Format Translators - Detects request formats and translates payloads to match the specific requirements of different LLM providers.
• Fallback Configurations - Implements failover logic to redirect requests to backup AI models when primary models fail.
• Usage Quotas - Enforces token-based usage quotas and balances for consumers interacting with AI models.
• Response Caching - Retrieves previous model responses using string or vector matching to lower API costs and latency.
• Semantic Caching - Stores model responses and dialogue context using semantic matching to reduce token usage and latency.
• Request Path Rewriters - Modifies original request paths and host headers to correctly route traffic to backend services.
• MCP Protocol Adapters - Transforms external REST APIs into protocol servers using templates to map tool calls to API endpoints.
• API Gateways - Implements a cloud-native API gateway compatible with both Kubernetes Ingress and Gateway API standards.
• AI Traffic Routing - Manages load balancing, fallback, and rate limiting for traffic routed to multiple model providers.
• API Traffic Management - Manages microservice entry points with advanced routing, rate limiting, and automated TLS certificates.
• Routing Policy Resources - Uses Kubernetes custom resource definitions to declaratively define routing and traffic policies within a cluster.
• Canary Deployment Controllers - Gradually shifts traffic to new versions using weight percentages to minimize deployment risk.
• Load Balancing Algorithms - Allows the selection and configuration of algorithms used to balance traffic across available backend nodes.
• Microservice Traffic Management - Implements cloud-native traffic management including canary releases, load balancing, and rate limiting for microservice architectures.
• Distributed Rate Limiting - Synchronizes request limits across multiple instances using a shared data store for consistent global throttling.
• Service Discovery - Dynamically locates and routes traffic to backend service instances using multiple service registries.
• Dynamic Service Discovery - Integrates with registries like Nacos and Consul to automatically resolve logical service names to network addresses.
• Load Balancers - Distributes incoming requests across backend services using algorithms like round-robin and consistent hashing.
• Model Request Proxies - Routes requests to various AI service providers using a unified interface for chat and embedding tasks.
• Service Traffic Routing - Routes traffic to specific service versions using weights or cookies for gradual rollouts.
• Proxy Headers - Manages the addition of real-IP headers and removal of proxy headers before forwarding traffic.
• API Gateway - Maps incoming HTTP requests to various backend protocols like gRPC, Dubbo, and MCP using request templates.
• Service Discovery Integrations - Connects to upstream clusters using discovery mechanisms such as DNS, Kubernetes, or external service registries.
• Traffic Routing - Provides a console and configuration system to define how requests are directed to backend services.
• Traffic Routing Policies - Provides declarative routing policies using metadata to control network traffic behavior.
• Weighted Traffic Splitting - Distributes traffic to specific service versions based on weights or headers to enable gradual rollouts.
• API Access Security - Protects API endpoints using a combination of WAF filtering and authentication strategies like JWT and HMAC.
• Enterprise Identity Providers - Secures API access using OIDC, OAuth2, and JWT to manage enterprise-grade authentication and authorization policies.
• Usage Limits - Enforces AI model access quotas per consumer to ensure fair token distribution.
• Plugin-Based Logic Extensions - Uses a plugin-based architecture to execute custom logic and built-in extensions during the request lifecycle.
• Rate Limiting - Provides rate limiting based on dynamic attributes such as IP addresses, headers, or consumer identities.
• Service Discovery Registries - Connects to external discovery instances using server addresses and credentials to track service locations.
• Wasm-Based Plugins - Executes custom logic through WebAssembly modules to modify request and response handling without restarting the proxy.
• Agent API Gateways - Exposes autonomous agents as secure, observable, and scalable web services through an AI-specific API gateway.
• API Lifecycle Management Suites - Provides an integrated platform for end-to-end API deployment, including version control, canary releases, and automated rollbacks.
• Request and Response Transformers - Provides comprehensive transformation of request and response headers, query parameters, and body content.
• Protected Endpoints - Provides protected endpoints secured via WAF capabilities and identity providers including OIDC, JWT, and HMAC.
• HTTP Redirects - Handles HTTP redirection to send client requests to different target destinations.
• Intent Classification Pipelines - Categorizes incoming requests into predefined domains to determine the appropriate downstream model or cache.
• Skill Packaging - Bundles agent skills, configurations, and dependencies into versioned packages for portable distribution.
• Custom Agent Distributions - Provides mechanisms for bundling and publishing versioned agent packages to a discovery marketplace.
• Agent Installation - Provides web-based and CLI methods for deploying agent packages to local environments.
• Sandbox Environments - Creates hosted, sandboxed runtime environments for MCP servers including resource allocation and authentication.
• AI Access Control - Provides fine-grained authorization logic to restrict access to specific AI tools based on whitelists or role-based headers.
• Skill Marketplaces - Hosts AI skills in a searchable portal where users can filter and subscribe to specific capabilities.
• AI Monitoring - Provides observability tools to track real-time token counts and performance metrics per AI model provider.
• AI Provider Managers - Manages API keys for multiple AI vendors and implements health-based token degradation strategies.
• AI Tool Definitions - Automatically discovers and refreshes structured specifications for tools used by AI agents from registered servers.
• Conversation History Management - Caches dialogue based on request headers and injects previous turns into the prompt context.
• Model Comparison Tools - Executes a single prompt across multiple models simultaneously to compare response quality and accuracy.
• Private Context Integration - Integrates proprietary documents and knowledge bases to ground large language models in private domain-specific data.
• Prompt Engineering - Inserts custom text or dynamic data into prompts to optimize language model performance.
• Prompt Templates - Creates reusable request structures to ensure consistency across similar types of AI interactions.
• Query Optimization - Analyzes natural language queries and rewrites them into optimized keywords for better retrieval relevance.
• Web Search Integrations - Enables language models to query real-time web data to expand their knowledge base with current information.
• Agent Cataloging - Creates catalog entries for agents by defining names, types, and technical metadata for users.
• Agent Marketplaces - Enables users to discover and filter available AI agents and view their technical configurations via a portal.
• Server Discovery Portals - Lists configured MCP servers in a developer portal for user discovery and tool subscription.
• A/B Testing - Routes traffic to specific application versions based on request attributes to execute A/B tests.
• REST to GraphQL Translation - Translates incoming REST requests into GraphQL queries and forwards them to a GraphQL endpoint.
• Usage Billing Systems - Tracks resource consumption based on tokens or request counts to generate internal or commercial billing details.
• Developer Portals - Distributes configured API products to centralized developer portals for discovery and subscription.
• Gateway Installation Profiles - Provides predefined configuration profiles for installing the gateway and observability tools onto Kubernetes or Docker.
• Interactive API Explorers - Renders an interactive interface for visualizing and testing available API endpoints directly from the console.
• AI Artifact Catalogs - Distributes models and agents through a centralized artifact catalog featuring versioning and grayscale releases.
• Inter-Plugin State Sharing - Stores and retrieves metadata properties during a request lifecycle to pass state across different plugins.
• Backend Endpoint Definitions - Configures and registers the base URLs and addresses of backend services for targeted routing.
• Request Retries - Automatically retries failed requests to the next available upstream server based on defined conditions.
• Custom Resource Mappings - Translates Kubernetes custom resources and Gateway API definitions into active network filter configurations.
• Standalone Container Deployments - Provides a single-container image deployment for rapid local setup and testing of the gateway.
• Custom Resource Definitions - Creates custom API resources using protobuf definitions to extend configuration capabilities within Kubernetes.
• Custom Resource Controllers - Monitors changes to custom resource instances to trigger business logic and automate network filter generation.
• Deployment Scaling - Manages gateway scaling via manual replica counts or automatic CPU-based pod scaling.
• Progressive Delivery Strategies - Coordinates workload updates and routing rules to execute phased rollouts with automated approvals.
• Multi-Architecture Deployments - Supports installation across Kubernetes Operator mode, non-Kubernetes environments, and ARM architectures.
• OCI-Compliant Plugin Packaging - Packages WebAssembly binaries and metadata into OCI-compliant container images for standardized distribution and hot-loading.
• Rate Limiting Algorithms - Implements the Token Bucket algorithm to allow traffic bursts while maintaining a steady request rate.
• Plugin Development Kits - Provides a development kit with wrappers for context management to simplify the creation of custom gateway extensions.
• Scope Configurations - Assigns plugin settings at global, service, domain, or route levels to enable granular configuration overrides.
• Traffic Tagging - Adds specific request headers based on content matching to identify traffic for canary deployments and testing.
• Version Upgrades - Provides tools for updating existing gateway installations to newer versions without losing configuration.
• Registry-Based Service Discovery - Forwards requests to microservices by resolving addresses from external registries like Nacos or Consul.
• gRPC Traffic Routing - Directs requests to backend services using specialized protocols such as gRPC and HTTP/2.
• HTTP Traffic Manipulation Tools - Provides tools for intercepting and modifying HTTP headers and body content for both requests and responses in real-time.
• Load Balancer IP Address Management - Configures listening ports, load balancer IP addresses, and source IP ranges for incoming network traffic.
• Network Port Configuration - Allows custom configuration of the network ports the gateway uses for listening to HTTP and HTTPS traffic.
• Backend Protocol Settings - Provides configurations for specifying the communication protocols, such as HTTP or gRPC, used between the gateway and upstream services.
• Protocol-Agnostic Proxies - Routes client requests to backend servers using streaming protocols while managing the connection lifecycle.
• Proxy Servers - Routes requests to native services and maintains sessions to facilitate communication with specialized data sources.
• Dubbo Integration - Transforms standard HTTP requests into Dubbo protocol calls to trigger remote service methods.
• Layer 4 TCP Routing - Directs raw TCP network traffic to backend services using a layer 4 routing mechanism.
• Traffic Shadowing - Duplicates a percentage of live traffic to a shadow service for auditing and testing.
• Transport Timeouts - Configures idle timeouts and buffer limits for both downstream clients and upstream service connections.
• AI Security - Filters sensitive information and performs security checks specifically on requests directed at AI resources.
• Mutual TLS Authentication - Implements bidirectional identity verification via mutual TLS between clients and the gateway.
• API Credential Managers - Generates and manages unique API keys and secrets for different environments to maintain multiple credential sets.
• API Key Authentication - Validates clients by matching API keys from request headers or URL parameters against authorized consumers.
• Bot Blocking - Identifies and blocks web crawlers and scrapers based on User-Agent patterns.
• Authentication Scope Management - Controls which requests require identity verification using path-based whitelist or blacklist rules.
• Basic Authentication - Validates users based on the HTTP Basic Auth standard and attaches identity to the request header.
• Communication Encryption - Secures the communication channel between the gateway and backend services using client certificates and SNI.
• AI Content Filters - Scans and filters AI model prompts and responses to block harmful content and ensure compliance.
• Content Moderation - Inspects model inputs and outputs via security services to block non-compliant or inappropriate content.
• SSL/TLS Certificate Management - Automates the provisioning and renewal of SSL/TLS certificates through Let's Encrypt.
• Request Access Restrictions - Implements per-request authorization logic to restrict domain access to authorized JWT consumers.
• External Identity Provider Integration - Authenticates users via OIDC or OAuth2 providers to automatically create or link developer accounts.
• TLS Certificate Management - Simplifies the provisioning and renewal of security certificates for secure traffic.
• Security Policy Scoping - Allows security policies to be applied at global, domain, or route levels for precise traffic control.
• HTTP Request Filtering - Blocks suspicious traffic using a rule-based engine integrated with the OWASP Core Rule Set.
• External Authorization Proxies - Delegates permission validation to an external security service via proxy requests for centralized authorization.
• JWT Authentication - Validates JSON Web Tokens from headers or cookies using public keys or symmetric secrets.
• Mutual TLS Configurations - Configures TLS cipher suites and CA certificates to secure the connection between clients and the gateway.
• Mutual TLS Transports - Establishes mutual TLS and verifies SNI for encrypted traffic between the gateway and upstream services.
• OIDC Authentication Plugins - Verifies user identities using the OpenID Connect protocol to manage the full authentication lifecycle.
• Policy-Based Access Control - Enforces security permissions by evaluating incoming requests against external Open Policy Agent rules.
• Model Inference Metrics - Tracks model inference metrics including token usage and request latency across gateways and routes.
• Request Authentication - Verifies caller identity by validating a cryptographic HMAC signature generated from a shared secret.
• Dynamic Data Masking - Replaces sensitive words in request and response bodies using regex and grok patterns to prevent data leaks.
• PII Masking - Encrypts or masks sensitive information in requests and responses to prevent PII leaks.
• Service Communication Security - Defines trust domains and root certificates to manage identities and encryption for inter-service communication.
• SSL Certificate Automation - Automates the issuance and renewal of SSL certificates using ACME providers.
• TLS Management - Configures minimum and maximum TLS protocol versions and cipher suites for secure communication.
• Token Signature Verification - Validates incoming JWT tokens against configured consumers to determine if a request is permitted.
• OAuth2 Access Token Issuance - Generates JWT-based access tokens using the client credentials grant flow to authenticate service callers.
• Traffic Filtering - Inspects request headers and bodies using predefined security rule sets to block attacks.
• Service Call Fallbacks - Forwards traffic to a backup service when the primary backend is unavailable or returns errors.
• Hierarchical Configuration Overrides - Applies configuration settings across global, service, domain, and route levels using a hierarchical override system.
• Plugin Execution Ordering - Allows precise control over the order of plugin execution within the filter chain using phases and priority values.
• Static Endpoint Mapping - Directs requests to backend services using fixed IP addresses or DNS domain names.
• URL-Based Plugin Loading - Fetches Wasm plugins from remote URLs via HTTP or HTTPS to extend functionality without rebuilding images.
• Capability Packaging - Wraps models, agents, and servers into standardized product packages with custom metadata and guides.
• Plugin Configuration Defaults - Implements a tiered configuration system where specific plugin settings override global defaults to control traffic.
• Execution Scoping - Applies plugins based on match rules for domains or routes to control exactly where logic is executed.
• Traffic Modification Plugins - Provides Wasm plugins to intercept and alter HTTP headers, bodies, and TCP data frames.
• API Gateways - Functions as an API gateway that extends request processing logic using hot-loadable WebAssembly plugins.
• Gateway Observability - Provides integrated dashboards to track gateway traffic, system health, and overall performance.
• IP Access Restrictions - Blocks or permits incoming requests based on IP address whitelists or blacklists.
• Log Statistics Visualizers - Displays access logs through a centralized monitoring dashboard to visualize and analyze traffic patterns.
• LLM Performance Monitoring - Tracks LLM performance metrics, including token usage, success rates, and response times.
• AI Workload Dashboards - Offers global visualization dashboards to monitor AI workload health, API call trends, and traffic patterns.
• Request Traffic Monitors - Generates dashboards to track request volume, processing duration, and success rates across API endpoints.
• Observability Suites - Integrates and configures comprehensive toolsets for collecting and analyzing logs, metrics, and traces.
• Proxy Management Dashboards - Provides a web-based console for configuring network traffic routing, access policies, and security protocols.
• Request Timeouts - Defines the maximum duration the gateway waits for a response before timing out a request.
• Concurrency Limiters - Caps the number of simultaneous requests to backend services to prevent resource exhaustion and system crashes.
• Traffic Interception Tools - Uses Wasm-compiled logic to intercept and modify HTTP traffic during the request lifecycle.
• Traffic Monitoring Tools - Collects and visualizes network traffic metrics to verify routing and audit user traffic metadata.
• Token Usage Analytics - Tracks real-time input and output token consumption per model provider to enforce quotas and billing.
• Token Consumption Trackers - Extracts and aggregates input and output token counts per second to analyze model consumption.
• HTTP Header Manipulators - Provides utilities for adding, removing, or modifying HTTP header fields in both incoming requests and outgoing responses.
• Tool Parameter Definitions - Defines data types and placement for AI tool arguments across URL paths, query strings, and request bodies.
• Request Body Validations - Verifies request bodies and headers against JSON Schema or OAS3 definitions to reject malformed data.
• AI Output Validation - Formats non-streaming AI outputs into valid JSON and validates them against defined schemas.
• AI Response Templates - Converts raw HTTP responses into AI-friendly formats using a template engine with conditional logic.
• RPC Service Mappers - Maps incoming HTTP paths and methods to specific internal service interfaces for RPC communication.
• Search Result Injection - Retrieves live search engine data and incorporates it into model prompts to improve response accuracy.
• Encrypted Backend Routing - Implements secure communication with backend services using encrypted protocols including HTTPS and GRPCS.
• Frontend Version Routing - Routes users to different frontend versions based on cookies or headers to support A/B testing.
• HTTP Response Handlers - Implements mechanisms to construct and return predefined HTTP responses, including custom status codes and bodies.
• Service Mesh And Gateways - Delivers a cloud-native API gateway based on Istio.

Star history