awesome-repositories.com
Blog
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectAboutHow we rankPressMCP server
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
·
BerriAI avatar

BerriAI/litellm

0
View on GitHub↗
50,579 stars·8,915 forks·Python·15 viewsdocs.litellm.ai/docs↗

Litellm

LiteLLM is a unified gateway and proxy server designed to centralize access to over one hundred language model providers. It provides a standardized API interface that abstracts vendor-specific schemas, allowing developers to interact with diverse models through a single, consistent format. By acting as a central traffic management layer, it enables organizations to route, secure, and govern model interactions across multiple deployments.

The platform distinguishes itself through its policy-driven architecture, which uses configuration-based routing to manage traffic distribution, load balancing, and automatic fallbacks without requiring code changes. It incorporates a robust security and compliance layer that enforces content moderation, secret redaction, and fine-grained access control. Additionally, it supports complex operational requirements such as semantic routing, rule-based complexity scoring, and persistent virtual key management for multi-tenant environments.

Beyond core routing, the project provides comprehensive governance and observability tools to monitor usage, track spending, and log request metadata across teams. It includes an integrated software development kit for tool calling and agent orchestration, alongside support for advanced features like response caching, batch processing, and structured output configuration. The system is designed for enterprise-wide deployment, offering features for audit logging, single sign-on integration, and granular cost reporting.

Features

  • Model Gateways - Provides a unified interface to access over one hundred different language models through a standard chat completion format.
  • Request Routers - Distributes model queries across various deployments based on cost, performance, and availability rules.
  • Text Generation APIs - Provides unified interfaces for generating text and code completions across multiple AI model providers.
  • Traffic Orchestrators - Routes incoming requests through a centralized gateway that manages authentication and load balancing.
  • Model Safety Filters - Implements safety filters to validate or block model inputs and outputs based on custom content policies.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI
  • AI Governance Tools - Enforces spending budgets, rate limits, and usage tracking across teams to control operational expenses.
  • Governance Proxies - Acts as a security and compliance layer that enforces usage limits, content guardrails, and cost tracking.
  • Provider Abstractions - Maps diverse vendor-specific API schemas into a single standardized interface for consistent interaction.
  • Tool Integration Frameworks - Connects external tools to language models by listing and executing functions through a standardized interface.
  • Content Guardrails - Intercepts request and response streams to apply content moderation and safety policy enforcement.
  • Security Proxies - Secures model interactions with centralized authentication, content guardrails, and audit logging.
  • Agent Gateways - Links autonomous agents to model providers through a centralized gateway to standardize communication.
  • Intelligent Routers - Distributes requests across different AI models based on performance, cost, or semantic relevance.
  • Model Accessors - Enables selection of specific AI models within function calls to leverage diverse capabilities.
  • Tool-Calling Frameworks - Enables AI models to define and execute custom tools with support for parallel function calls and workflow management.
  • Server Management Systems - Configures server connections using multiple transports while enforcing access control.
  • Software Development Kits - Provides a unified interface for interacting with diverse language models and external tool-calling services.
  • Access Control Systems - Secures administrative access using single sign-on, audit logging, and content guardrails.
  • Proxy Authentication - Secures proxy access using API keys and identity providers to manage user permissions and enforce access control.
  • Complexity-Based Routers - Classifies request complexity to send queries to appropriate models based on cost and performance.
  • Routing Configurations - Defines traffic distribution and model mapping rules through externalized configuration files.
  • Agent Runtimes - Executes registered agents using either a native SDK or a standard chat-completion interface.
  • Response Streamers - Enables real-time response chunking during completion requests for efficient text generation.
  • Semantic Routers - Routes incoming requests to specialized models by matching input content against examples using embedding-based similarity.
  • AI and Machine Learning - SDK and proxy for accessing multiple LLM APIs via a unified format.
  • AI Frameworks and SDKs - Unified API proxy for LLM providers with spend tracking.
  • AI Gateways - Unified API gateway for multiple LLM providers with spend tracking.
  • API And Routing - Unified library for calling multiple LLM APIs.
  • Application Frameworks - Proxy server and SDK to call multiple LLM APIs uniformly.
  • Development Frameworks - Unified API interface for calling various LLM providers.
  • Inference and Serving - Unified API proxy for multiple model providers.
  • Inference Engines - Unified API for calling various model providers.
  • Large Language Models - Unified interface for calling multiple LLM providers.
  • LLM Development Utilities - Gateway for calling multiple LLM APIs in a unified format.
  • Local Development and Serving - Standardize requests across multiple different LLM providers.
  • Machine Learning Operations - Unified interface to call all LLM APIs with consistent output formatting.
  • Model Serving and Inference - Proxy server to call 100+ LLM APIs with unified formatting.
  • Model Serving & Deployment - Acts as a proxy server for calling multiple LLM APIs.
  • Natural Language Processing - Unified gateway for calling various large language model APIs.
  • API Gateways and Proxies - Proxy server for standardizing calls to multiple LLM providers.
  • Large Language Models (LLMs) - Listed in the “Large Language Models (LLMs)” section of the The Incredible Pytorch awesome list.
  • Budget Management Systems - Manages multi-tenant environments by enforcing budgets and spending limits across teams and projects.
  • Response Caching - Stores and retrieves previous model outputs in a cache to reduce latency and operational costs.
  • Load Balancers - Distributes traffic across multiple model deployments while monitoring rate limits and token usage.
  • Access Control Policies - Enforces fine-grained access control and usage tracking by passing user identifiers in request headers.
  • API Key Management - Maintains stateful access control and usage tracking by mapping virtual API keys to specific budgets.
  • Provider Authenticators - Verifies requests to model providers by configuring required API keys.
  • Proxy Security Tools - Protects infrastructure using single sign-on, role-based access control, and secret management integrations.
  • Usage Limiters - Enforces spending budgets and rate limits per user or key to control costs and prevent service abuse.
  • Batch Processors - Executes batch processing tasks on supported providers via standard compatible API interfaces.
  • Prompt Caches - Stores large context or tool definitions to reduce token usage and improve performance.
  • Structured Output Converters - Transforms schema formats into provider-specific requirements to ensure structured data returns.
  • Content Validation - Filters model content using integrated moderation services with configurable thresholds.
  • Traffic Load Balancers - Distributes traffic across multiple model deployments using routing rules and automatic fallbacks for high availability.
  • Data Redaction Tools - Removes sensitive information like API keys from model requests automatically using secret detection guardrails.
  • OAuth Configurations - Sets up OAuth 2.0 for servers by using automatic provider discovery or explicit credentials.
  • Model Routing Configurations - Defines available models and provider-specific parameters within a configuration file.
  • Interaction Logs - Monitors agent interactions by viewing request content, user metadata, and performance metrics.
  • Observability Platforms - Captures and monitors request metadata, latency, and costs across all model interactions.
  • Agent Registrations - Adds various agent types into a centralized gateway to enable unified management.
  • Document Rerankers - Reorders lists of documents based on specific queries using provider reranking models.
  • Embedding Generators - Creates numerical representations for input strings to optimize retrieval performance.
  • Proxy Routers - Routes requests to specific model providers by defining model mappings and API credentials.
  • Spend Tracking Tools - Reports model usage costs by grouping requests with custom tags for granular spend analysis.
  • Virtual Key Managers - Generates virtual API keys with specific rate limits and usage tracking for managing user access.
  • Activity Monitors - Observes proxy activity with team-based logging and durable log exports to ensure compliance.
  • Observability Callbacks - Triggers asynchronous hooks to log request metadata and performance metrics to external services.
  • Request Logs - Captures and stores model request and response data for monitoring and auditing purposes.

    • Star history

    Star history chart for berriai/litellmStar history chart for berriai/litellm

    Open-source alternatives to Litellm

    Similar open-source projects, ranked by how many features they share with Litellm.
    • bentoml/openllmbentoml avatar

      bentoml/OpenLLM

      12,115View on GitHub↗

      OpenLLM is a framework for deploying, managing, and scaling open-source large language models

      Pythonbentomlfine-tuningllama
      View on GitHub↗12,115
    • open-webui/open-webuiopen-webui avatar

      open-webui/open-webui

      142,694View on GitHub↗

      Open WebUI is a self-hosted, web-based platform designed for interacting with local and remote artificial intelligence models. It functions as a unified interface and orchestration suite, enabling users to build, deploy, and manage specialized AI agents equipped with custom instructions, external tool access, and private knowledge bases. The platform distinguishes itself through a modular architecture that supports complex AI workflows. It features a plugin-based framework for custom logic and pipeline-based request processing, allowing developers to filter or transform data streams before th

      Pythonaillmllm-ui
      View on GitHub↗142,694
    • ggml-org/llama.cppggml-org avatar

      ggml-org/llama.cpp

      116,799View on GitHub↗

      Llama.cpp is an inference engine designed for the local execution of text-based and multimodal language models on consumer hardware. It provides a core environment for running models that process both text and image inputs, utilizing hardware-accelerated backends to optimize performance across diverse CPU and GPU architectures. The project distinguishes itself by offering a lightweight HTTP server that adheres to standard API specifications, enabling chat completion, embeddings, and reranking services. It includes a suite of tools for model quantization and conversion, which reduces memory us

      C++ggml
      View on GitHub↗116,799
    • langchain-ai/langchainlangchain-ai avatar

      langchain-ai/langchain

      139,458View on GitHub↗

      LangChain is an orchestration framework designed for building, managing, and deploying applications powered by large language models. It provides a unified integration layer that normalizes disparate model provider APIs into a consistent set of primitives, enabling developers to build complex, multi-step AI workflows that manage state, memory, and tool execution. The project distinguishes itself through a durable execution runtime that maintains persistent state across long-running processes by checkpointing progress to external storage. It models agent workflows as directed graphs, allowing

      Pythonagentsaiai-agents
      View on GitHub↗139,458
    See all 30 alternatives to Litellm→

    Frequently asked questions

    What does berriai/litellm do?

    LiteLLM is a unified gateway and proxy server designed to centralize access to over one hundred language model providers. It provides a standardized API interface that abstracts vendor-specific schemas, allowing developers to interact with diverse models through a single, consistent format. By acting as a central traffic management layer, it enables organizations to route, secure, and govern model interactions across multiple deployments.

    What are the main features of berriai/litellm?

    The main features of berriai/litellm are: Model Gateways, Request Routers, Text Generation APIs, Traffic Orchestrators, Model Safety Filters, AI Governance Tools, Governance Proxies, Provider Abstractions.

    What are some open-source alternatives to berriai/litellm?

    Open-source alternatives to berriai/litellm include: bentoml/openllm — OpenLLM is a framework for deploying, managing, and scaling open-source large language models. open-webui/open-webui — Open WebUI is a self-hosted, web-based platform designed for interacting with local and remote artificial intelligence… langchain-ai/langchain — LangChain is an orchestration framework designed for building, managing, and deploying applications powered by large… ggml-org/llama.cpp — Llama.cpp is an inference engine designed for the local execution of text-based and multimodal language models on… ollama/ollama — Ollama provides a framework for running and managing local machine learning models. It includes a command-line… vllm-project/vllm — vLLM is a high-throughput inference engine designed for the efficient serving and execution of large language models.…