LLM-Response-Caching-Proxies
For Caching-Layer für LLM-API-Aufrufe, the strongest matches are berriai/litellm (LiteLLM is a unified LLM API gateway and proxy), helicone/helicone (Helicone is a reverse-proxy AI gateway that intercepts LLM) and zilliztech/gptcache (GPTCache is a semantic caching middleware that reduces repeated). alibaba/higress is also worth a look. Each is ranked by relevance to your query, popularity and recent activity.
Open-Source-Middleware und Proxy-Tools, die Ausgaben von Sprachmodellen zwischenspeichern, um API-Latenz und Kosten zu reduzieren.
LLM-Response-Caching-Proxies
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- berriai/litellm
BerriAI/litellm
50,579Auf GitHub ansehenLiteLLM is a unified gateway and proxy server designed to centralize access to over one hundred language model providers. It provides a standardized API interface that abstracts vendor-specific schemas, allowing developers to interact with diverse models through a single, consistent format. By acting as a central traffic management layer, it enables organizations to route, secure, and govern model interactions across multiple deployments. The platform distinguishes itself through its policy-driven architecture, which uses configuration-based routing to manage traffic distribution, load balanc
LiteLLM is a unified LLM API gateway and proxy that includes prompt caching, cost tracking, and transparent request routing, directly meeting the need to cache responses and reduce repeated API calls while tracking spending.
PythonResponse Caching - helicone/helicone
Helicone/helicone
5,830Auf GitHub ansehenHelicone is an AI gateway and observability platform designed to intercept, manage, and monitor interactions with large language models. By acting as a reverse-proxy, it provides a centralized layer for routing requests across multiple AI providers, allowing developers to maintain consistent application logic while gaining deep visibility into model performance, usage, and costs. The platform distinguishes itself through a robust suite of traffic management and prompt engineering tools. It enables policy-driven control, including automatic failover between providers, rate limiting, and edge-b
Helicone is a reverse-proxy AI gateway that intercepts LLM API calls and includes response caching (tagged as Execution Result Caches) alongside cost tracking, rate limiting, and transparent proxying—directly covering the caching, cost reduction, and transparent-proxy requirements you asked for.
TypeScriptResponse CachingModel Request Proxies - zilliztech/gptcache
zilliztech/GPTCache
8,068Auf GitHub ansehenGPTCache is a semantic caching layer and response optimizer for large language models. It functions as pluggable middleware for orchestration frameworks, utilizing vector database caching to store and retrieve model responses based on the semantic similarity of prompts rather than exact text matches. The system uses embeddings to determine cache hits by comparing the distance between new queries and stored vectors. It employs a hybrid storage model that persists original prompts in relational databases while maintaining high-dimensional embeddings in vector stores. The project covers a broad
GPTCache is a semantic caching middleware that reduces repeated LLM API calls by caching responses based on prompt similarity, which aligns well with your goal of a caching proxy, though cost tracking and explicit TTL configuration are not highlighted.
PythonResponse Caching - alibaba/higress
alibaba/higress
7,558Auf GitHub ansehenHigress is an AI API gateway and cloud-native traffic manager that functions as a Kubernetes ingress controller. It provides a centralized system for routing, securing, and optimizing traffic directed toward large language models, AI agents, and microservice architectures. The project distinguishes itself through deep AI orchestration, including the ability to host and manage Model Context Protocol servers that transform REST APIs into tools for AI agents. It features specialized AI infrastructure for model request proxying, protocol translation across multiple providers, and semantic-based c
Higress is an AI API gateway that proxies and caches requests to large language models, directly addressing the need to reduce repeated API calls, though its broader traffic-management features go beyond a dedicated caching proxy.
GoResponse CachingModel Request Proxies
