Wei-Shaw/claude-relay-service
Claude Relay Service
This project is a secure intermediary proxy gateway for large language model APIs. It functions as a relay service that forwards requests to AI providers while managing service accounts and routing traffic.
The service provides a compatibility layer that supports multiple endpoint formats, allowing different third-party AI clients to communicate with a single provider. It distinguishes itself through a service account management system that assigns individual proxy settings to multiple accounts to prevent IP bans and distributes traffic via load balancing to avoid rate limits.
The system includes a rate limiter that restricts access based on token volume, concurrency, and custom identification keys. It monitors usage through a tracking system that records token consumption and request metrics per user. Reliability is maintained through a circuit-breaker mechanism that detects upstream connection failures and pauses routing to affected accounts using cooldown timers.
Features
- Request Relays - Provides a secure intermediary relay layer that forwards incoming requests to remote AI providers.
- AI Gateway Management - Provides centralized management of AI proxy settings, request rates, and token usage monitoring.
- LLM Gateways - Functions as a secure LLM API proxy gateway that aggregates multiple providers and manages service accounts.
- AI Model Load Balancers - Distributes AI API traffic across multiple service accounts to optimize rate limits and handle upstream failures.
- AI Provider Proxies - Routes requests to Claude AI through a secure intermediary to manage service accounts and prevent IP bans.
- Proxy Routing Services - Routes incoming traffic to specific service accounts based on account health and configured proxy settings.
- Usage-Based Access Restrictions - Restricts API access based on request rates, token volume, concurrent connections, and client identifiers.
- Service Account Management - Implements a system for managing non-human service accounts and their associated proxy settings to avoid IP bans.
- Rate Limiting - Enforces request and volume quotas based on token consumption metrics against unique user access keys.
- Token Usage Analytics - Tracks and records LLM token consumption and request counts per user for monitoring and access control.
- API Compatibility Layers - Exposes multiple endpoint formats via a compatibility layer to ensure different AI clients can communicate with one provider.
- Third-Party Service Integrations - Connects diverse AI clients to a centralized relay service using custom access keys and compatible formats.
- Circuit Breakers - Implements a circuit-breaker mechanism that pauses routing to failing upstream accounts using cooldown timers.
- Circuit Breakers - Provides a circuit-breaker mechanism that detects upstream connection failures and pauses routing to affected accounts using cooldown timers.
- Stateless Architectures - Employs a stateless architecture to forward AI requests without storing conversation content on the server.
- Context Usage Monitors - Provides a monitoring dashboard to track token consumption and request metrics per user.
- Endpoint Translation Maps - Translates various third-party API request formats into a single standardized internal format for upstream delivery.
- Format Compatibility Layers - Ships an endpoint prefixing system that ensures compatible communication between various third-party AI clients and providers.
- AI Tools - Relay service for accessing Claude API in restricted regions.
