This project is a secure intermediary proxy gateway for large language model APIs. It functions as a relay service that forwards requests to AI providers while managing service accounts and routing traffic.
The service provides a compatibility layer that supports multiple endpoint formats, allowing different third-party AI clients to communicate with a single provider. It distinguishes itself through a service account management system that assigns individual proxy settings to multiple accounts to prevent IP bans and distributes traffic via load balancing to avoid rate limits.
The system includes a rate limiter that restricts access based on token volume, concurrency, and custom identification keys. It monitors usage through a tracking system that records token consumption and request metrics per user. Reliability is maintained through a circuit-breaker mechanism that detects upstream connection failures and pauses routing to affected accounts using cooldown timers.
