BerriAI/litellm

Features

• Model Gateways - Provides a unified interface to access over one hundred different language models through a standard chat completion format.
• Request Routers - Distributes model queries across various deployments based on cost, performance, and availability rules.
• Text Generation APIs - Provides unified interfaces for generating text and code completions across multiple AI model providers.
• Traffic Orchestrators - Routes incoming requests through a centralized gateway that manages authentication and load balancing.
• Model Safety Filters - Implements safety filters to validate or block model inputs and outputs based on custom content policies.
• AI Governance Tools - Enforces spending budgets, rate limits, and usage tracking across teams to control operational expenses.
• Governance Proxies - Acts as a security and compliance layer that enforces usage limits, content guardrails, and cost tracking.
• Provider Abstractions - Maps diverse vendor-specific API schemas into a single standardized interface for consistent interaction.
• Tool Integration Frameworks - Connects external tools to language models by listing and executing functions through a standardized interface.
• Content Guardrails - Intercepts request and response streams to apply content moderation and safety policy enforcement.
• Security Proxies - Secures model interactions with centralized authentication, content guardrails, and audit logging.
• Agent Gateways - Links autonomous agents to model providers through a centralized gateway to standardize communication.
• Intelligent Routers - Distributes requests across different AI models based on performance, cost, or semantic relevance.
• Model Accessors - Enables selection of specific AI models within function calls to leverage diverse capabilities.
• Tool-Calling Frameworks - Enables AI models to define and execute custom tools with support for parallel function calls and workflow management.
• Server Management Systems - Configures server connections using multiple transports while enforcing access control.
• Software Development Kits - Provides a unified interface for interacting with diverse language models and external tool-calling services.
• Access Control Systems - Secures administrative access using single sign-on, audit logging, and content guardrails.
• Proxy Authentication - Secures proxy access using API keys and identity providers to manage user permissions and enforce access control.
• Complexity-Based Routers - Classifies request complexity to send queries to appropriate models based on cost and performance.
• Routing Configurations - Defines traffic distribution and model mapping rules through externalized configuration files.
• Agent Runtimes - Executes registered agents using either a native SDK or a standard chat-completion interface.
• Response Streamers - Enables real-time response chunking during completion requests for efficient text generation.
• Semantic Routers - Routes incoming requests to specialized models by matching input content against examples using embedding-based similarity.
• AI and Machine Learning - SDK and proxy for accessing multiple LLM APIs via a unified format.
• AI Frameworks and SDKs - Unified API proxy for LLM providers with spend tracking.
• AI Gateways - Unified API gateway for multiple LLM providers with spend tracking.
• API And Routing - Unified library for calling multiple LLM APIs.
• Application Frameworks - Proxy server and SDK to call multiple LLM APIs uniformly.
• Development Frameworks - Unified API interface for calling various LLM providers.
• Inference and Serving - Unified API proxy for multiple model providers.
• Inference Engines - Unified API for calling various model providers.
• Large Language Models - Unified interface for calling multiple LLM providers.
• LLM Development Utilities - Gateway for calling multiple LLM APIs in a unified format.
• Local Development and Serving - Standardize requests across multiple different LLM providers.
• Machine Learning Operations - Unified interface to call all LLM APIs with consistent output formatting.
• Model Serving and Inference - Proxy server to call 100+ LLM APIs with unified formatting.
• Model Serving & Deployment - Acts as a proxy server for calling multiple LLM APIs.
• Natural Language Processing - Unified gateway for calling various large language model APIs.
• API Gateways and Proxies - Proxy server for standardizing calls to multiple LLM providers.
• Large Language Models (LLMs) - Listed in the “Large Language Models (LLMs)” section of the The Incredible Pytorch awesome list.
• Budget Management Systems - Manages multi-tenant environments by enforcing budgets and spending limits across teams and projects.
• Response Caching - Stores and retrieves previous model outputs in a cache to reduce latency and operational costs.
• Load Balancers - Distributes traffic across multiple model deployments while monitoring rate limits and token usage.
• Access Control Policies - Enforces fine-grained access control and usage tracking by passing user identifiers in request headers.
• API Key Management - Maintains stateful access control and usage tracking by mapping virtual API keys to specific budgets.
• Provider Authenticators - Verifies requests to model providers by configuring required API keys.
• Proxy Security Tools - Protects infrastructure using single sign-on, role-based access control, and secret management integrations.
• Usage Limiters - Enforces spending budgets and rate limits per user or key to control costs and prevent service abuse.
• Batch Processors - Executes batch processing tasks on supported providers via standard compatible API interfaces.
• Prompt Caches - Stores large context or tool definitions to reduce token usage and improve performance.
• Structured Output Converters - Transforms schema formats into provider-specific requirements to ensure structured data returns.
• Content Validation - Filters model content using integrated moderation services with configurable thresholds.
• Traffic Load Balancers - Distributes traffic across multiple model deployments using routing rules and automatic fallbacks for high availability.
• Data Redaction Tools - Removes sensitive information like API keys from model requests automatically using secret detection guardrails.
• OAuth Configurations - Sets up OAuth 2.0 for servers by using automatic provider discovery or explicit credentials.
• Model Routing Configurations - Defines available models and provider-specific parameters within a configuration file.
• Interaction Logs - Monitors agent interactions by viewing request content, user metadata, and performance metrics.
• Observability Platforms - Captures and monitors request metadata, latency, and costs across all model interactions.
• Agent Registrations - Adds various agent types into a centralized gateway to enable unified management.
• Document Rerankers - Reorders lists of documents based on specific queries using provider reranking models.
• Embedding Generators - Creates numerical representations for input strings to optimize retrieval performance.
• Proxy Routers - Routes requests to specific model providers by defining model mappings and API credentials.
• Spend Tracking Tools - Reports model usage costs by grouping requests with custom tags for granular spend analysis.
• Virtual Key Managers - Generates virtual API keys with specific rate limits and usage tracking for managing user access.
• Activity Monitors - Observes proxy activity with team-based logging and durable log exports to ensure compliance.
• Observability Callbacks - Triggers asynchronous hooks to log request metadata and performance metrics to external services.
• Request Logs - Captures and stores model request and response data for monitoring and auditing purposes.

Historique des stars