diegosouzapw/OmniRoute
OmniRoute
OmniRoute is a unified LLM API gateway that connects multiple AI providers to a single endpoint. Its primary purpose is to simplify the integration of various AI models into tools and agents by translating different provider formats into a standardized API.
The project distinguishes itself through a multi-strategy request routing system that optimizes for cost, speed, and availability, including automatic model fallbacks and a circuit-breaker resilience model to isolate provider failures. It employs a local-first security posture, using AES-256-GCM encryption to store API keys and conversation history on the user's own hardware. To further minimize costs, it aggregates free provider tiers and applies stacked semantic prompt compression to reduce token consumption.
The gateway also covers advanced traffic management via TLS fingerprint masking and proxy routing to bypass geo-restrictions. It provides a protocol-based agent gateway, allowing AI agents to autonomously manage routing and provider configurations, and includes hybrid vector-keyword memory for context-aware recall of conversation history.
Administrative control is available via a command line interface for managing providers, routing rules, and remote instances using scoped access tokens.
Features
- LLM Gateways - Provides a unified API gateway that aggregates multiple LLM providers into a single standardized endpoint.
- Unified AI API Endpoints - Provides a single OpenAI-compatible endpoint that abstracts multiple AI providers into a unified interface.
- AI Model Routing Strategies - Distributes prompts across providers using logic like cost-optimization, priority-based fallbacks, and round-robin.
- OpenAI-Compatible APIs - Provides an OpenAI-compatible endpoint to integrate a wide range of AI tools without manual configuration.
- AI Request Routing - Routes AI requests across providers using cost-optimization and priority-based fallbacks to ensure uptime.
- API Translation Layers - Maps proprietary model formats to standard API specifications, allowing multiple AI models to share one endpoint.
- Conversation Memory Stores - Persists interaction history to maintain state and context across multi-turn AI conversations.
- LLM Cost Management - Optimizes token usage and operational costs through semantic prompt compression and free tier aggregation.
- Model Fallbacks - Automatically switches to alternative AI models when the primary provider fails or hits rate limits.
- Sampling Provider Fallbacks - Provides instant failover to backup providers when the primary AI service is unavailable or rate-limited.
- Provider Failover Handlers - Implements millisecond failover to alternative providers when primary quotas are exhausted or services fail.
- LLM Token Compression - Reduces billed token usage through semantic pruning and deduplication of prompts to maximize the context window.
- Conversation Memory Retrieval - Enables context-aware recall of conversation history using a hybrid of keyword search and vector retrieval.
- Local Database Security - Implements local database security to ensure the privacy of encrypted API keys and usage history.
- Local-First Storage - Implements a local-first storage strategy using an encrypted database to ensure user data sovereignty and privacy.
- Encrypted - Saves API keys and usage history in an AES-256-GCM encrypted database residing entirely on the user's hardware.
- Routing Strategies - Selects the next provider based on specific logic such as priority, round-robin, or cost-optimization.
- Vector Memory Stores - Uses hybrid vector and keyword indexing to store and retrieve long-term context for AI agents.
- API Protocol Translations - Transparently converts requests and responses between different AI provider formats to ensure cross-service compatibility.
- Model Connectivity Resilience - Ensures high availability of AI APIs through a multi-layer resilience model including circuit breakers and cooldowns.
- Credential Encryption - Secures API keys and authentication tokens using AES-256-GCM encryption when stored on the local disk.
- Encrypted Storage - Stores sensitive API keys and conversation history in an AES-256-GCM encrypted database on local hardware.
- Encrypted Credential Storage - Secures API keys and conversation history in an AES-256-GCM encrypted local-first database.
- Multi-Provider Credential Stores - Manages API key storage and granular access scopes for multiple AI providers in a centralized store.
- AI Provider Gateways - Provides a specialized gateway for AI agents to autonomously manage routing, providers, and data compression.
- Provider Circuit Breakers - Implements failure isolation for external AI provider calls using circuit breakers to allow for service recovery.
- Provider-Agnostic LLM Routing - Directs LLM prompts to multiple providers using automatic scoring for availability, speed, and cost.
- Agent Communication Protocols - Runs standardized servers that facilitate communication between agents using industry-standard AI protocols.
- Agent Gateways - Provides a unified interface for AI agents to autonomously manage routing and provider configurations.
- Configuration Utilities - Offers a one-command setup to route multiple coding agents through the unified gateway.
- AI Agent Servers - Implements a standardized server that exposes routing and provider configurations to AI agents.
- MCP Server Integrations - Serves tools over multiple transport layers using the Model Context Protocol for external integration.
- Free AI Tier Aggregators - Integrates with numerous free provider tiers to access AI capabilities without requiring a credit card.
- Free Tier Aggregators - Aggregates free service tiers from multiple AI providers into a single dashboard to maximize available resources.
- Token Quota Managers - Provides systems to track and maximize the consumption of free token quotas across multiple AI providers.
- MCP Gateways - Integrates Model Context Protocol gateways to allow AI agents to manage routing and memory autonomously.
- Tool Output Compression - Implements specialized compression for tool call results to minimize token consumption and reduce processing costs.
- Quota Aggregators - Combines documented free tiers from various provider pools to track and maximize total available tokens.
- AI - Tests how requests flow through different strategy combinations without sending actual data to a provider.
- A2A Protocol Servers - Runs a server using the A2A protocol to enable smart routing and health reporting between agents.
- CLI Gateways - Provides a command-line interface to manage providers, routing rules, and token compression settings.
- Local Reverse Proxies - Operates as a local proxy on the user's hardware to prevent sensitive data from leaving the machine.
- Routing Decision Audit Trails - Maintains a searchable audit trail of routing decisions and tool calls for operational observability.
- AI Content Filters - Employs automated guardrails to scan and filter AI model prompts and responses for harmful content.
- LLM Prompt Injection Prevention - Inspects LLM requests using a dedicated security guard to prevent prompt injection attacks.
- CLI System Management - Allows configuration of provider keys and routing rules directly from the terminal.
- Cost and Token Trackers - Tracks token consumption and calculates costs per session, accounting for cache hits and compression.
- Endpoint Tool Mappers - Maps diverse AI provider endpoints into a unified format for seamless integration with external tools.
- LLM Applications - Self-hostable AI gateway with automatic fallback routing.
