zilliztech/GPTCache
GPTCache
GPTCache is a semantic caching layer and response optimizer for large language models. It functions as pluggable middleware for orchestration frameworks, utilizing vector database caching to store and retrieve model responses based on the semantic similarity of prompts rather than exact text matches.
The system uses embeddings to determine cache hits by comparing the distance between new queries and stored vectors. It employs a hybrid storage model that persists original prompts in relational databases while maintaining high-dimensional embeddings in vector stores.
The project covers a broad range of capabilities, including semantic embedding generation, vector indexing, and policy-based cache eviction. It also provides user session isolation to prevent data leakage and can be hosted as a standalone cache server to handle requests over HTTP.
Features
- Semantic Caching Systems - Stores and retrieves LLM responses based on the semantic similarity of input queries to optimize performance.
- Embedding Generators - Converts text requests into vector representations to enable semantic similarity searches within the cache.
- Latency Reduction Techniques - Minimizes response latency by retrieving cached answers instead of triggering full model generation.
- LLM Cost Management - Reduces API expenses by reusing previously generated responses for semantically similar queries.
- Text Embedding Generators - Transforms user queries into dense vectors via external providers to facilitate context similarity searches.
- Vector Databases - Uses vector databases to store embeddings and match new queries against previously generated model outputs.
- Vector Retrieval Systems - Employs vector-based retrieval to match new queries against stored embeddings for semantic context retrieval.
- API Response Optimizations - Optimizes API performance and costs by reusing cached responses for semantically similar queries.
- Hybrid Storage Engines - Utilizes a hybrid storage architecture to maintain both relational metadata and vector embeddings.
- Response Caching - Saves previous AI model answers to eliminate redundant API calls and reduce latency.
- Vector Database Integrations - Provides connectors for integrating specialized vector databases to store and retrieve semantic embeddings.
- Relational Vector Engines - Combines relational databases for raw text and metadata with vector stores for high-dimensional embeddings.
- Vector Storage - Persists model responses and high-dimensional embeddings using specialized vector storage systems.
- Distance Metrics - Uses mathematical distance functions to compare query embeddings and determine semantic cache hits.
- AI Model Adapters - Provides a standardized adapter interface to connect the caching layer with various language and image models.
- Orchestration Store Integrations - Offers a pluggable caching backend compatible with common LLM orchestration frameworks like LangChain and LlamaIndex.
- LLM Performance Optimization Libraries - Optimizes the execution speed and efficiency of LLM chains through a semantic caching layer.
- Cache Hit Evaluation - Matches new queries against cached responses using exact string matching or embedding distance metrics.
- Model Abstractions - Implements a standardized interface to normalize diverse AI model API responses and request handling.
- Query Optimization - Enhances retrieval in LLM frameworks by caching common queries and their corresponding responses.
- Cache Eviction Policies - Implements predefined replacement rules to remove outdated or infrequently accessed entries when storage limits are reached.
- Vector Indexing - Organizes high-dimensional embeddings in vector databases to ensure rapid retrieval of similar queries.
- Similarity Thresholds - Uses distance metric thresholds to determine if a new request is semantically similar enough to a cached entry.
- Inference Session Isolation - Isolates cached entries using unique user identifiers to prevent data leakage and maintain context relevance.
- Unified Model Wrappers - Standardizes request protocols and APIs across different LLMs and multimodal services for consistent interaction.
- Application Services - Semantic caching library for LLM queries.
- Development Frameworks - Caching library to reduce latency and costs for LLM queries.
- Infrastructure and Utilities - Provides semantic caching to reduce API costs and latency.
- Memory And Caching - Semantic caching library to reduce API costs and latency.
- RAG Frameworks - Semantic caching layer to accelerate LLM response times.
- Data Storage Systems - Creates semantic caches for LLM query optimization.
- Development Packages - Semantic caching layer for LLM queries.
