This project is an LLM API proxy gateway and compatibility layer designed to route, translate, and proxy requests between model clients and various large language model providers. It functions as a multi-provider router that maps model requests to alternative backends based on configurable tiers and capabilities.
The gateway acts as a translation layer that converts API request and response formats between different providers, such as OpenAI or Gemini, to ensure compatibility. It includes a tool-use proxy to handle the execution and processing of model tool definitions and function calls, and it supports an interface for translating and forwarding incremental streaming responses to reduce latency.
The system provides model provider abstraction, allowing the switching of backends without changing client application code. It manages conversation context tracking and utilizes a model-to-model mapping table to redirect requests based on model size and capability.
