zilliztech/GPTCache

Features

• Semantic Caching Systems - Stores and retrieves LLM responses based on the semantic similarity of input queries to optimize performance.
• Embedding Generators - Converts text requests into vector representations to enable semantic similarity searches within the cache.
• Latency Reduction Techniques - Minimizes response latency by retrieving cached answers instead of triggering full model generation.
• LLM Cost Management - Reduces API expenses by reusing previously generated responses for semantically similar queries.
• Text Embedding Generators - Transforms user queries into dense vectors via external providers to facilitate context similarity searches.
• Vector Databases - Uses vector databases to store embeddings and match new queries against previously generated model outputs.
• Vector Retrieval Systems - Employs vector-based retrieval to match new queries against stored embeddings for semantic context retrieval.
• API Response Optimizations - Optimizes API performance and costs by reusing cached responses for semantically similar queries.
• Hybrid Storage Engines - Utilizes a hybrid storage architecture to maintain both relational metadata and vector embeddings.
• Response Caching - Saves previous AI model answers to eliminate redundant API calls and reduce latency.
• Vector Database Integrations - Provides connectors for integrating specialized vector databases to store and retrieve semantic embeddings.
• Relational Vector Engines - Combines relational databases for raw text and metadata with vector stores for high-dimensional embeddings.
• Vector Storage - Persists model responses and high-dimensional embeddings using specialized vector storage systems.
• Distance Metrics - Uses mathematical distance functions to compare query embeddings and determine semantic cache hits.
• AI Model Adapters - Provides a standardized adapter interface to connect the caching layer with various language and image models.
• Orchestration Store Integrations - Offers a pluggable caching backend compatible with common LLM orchestration frameworks like LangChain and LlamaIndex.
• LLM Performance Optimization Libraries - Optimizes the execution speed and efficiency of LLM chains through a semantic caching layer.
• Cache Hit Evaluation - Matches new queries against cached responses using exact string matching or embedding distance metrics.
• Model Abstractions - Implements a standardized interface to normalize diverse AI model API responses and request handling.
• Query Optimization - Enhances retrieval in LLM frameworks by caching common queries and their corresponding responses.
• Cache Eviction Policies - Implements predefined replacement rules to remove outdated or infrequently accessed entries when storage limits are reached.
• Vector Indexing - Organizes high-dimensional embeddings in vector databases to ensure rapid retrieval of similar queries.
• Similarity Thresholds - Uses distance metric thresholds to determine if a new request is semantically similar enough to a cached entry.
• Inference Session Isolation - Isolates cached entries using unique user identifiers to prevent data leakage and maintain context relevance.
• Unified Model Wrappers - Standardizes request protocols and APIs across different LLMs and multimodal services for consistent interaction.
• Application Services - Semantic caching library for LLM queries.
• Development Frameworks - Caching library to reduce latency and costs for LLM queries.
• Infrastructure and Utilities - Provides semantic caching to reduce API costs and latency.
• Memory And Caching - Semantic caching library to reduce API costs and latency.
• Data Storage Systems - Creates semantic caches for LLM query optimization.
• Development Packages - Semantic caching layer for LLM queries.

Star history