alibaba/zvec

Features

• Hybrid Search Engines - Functions as a hybrid search engine integrating vector-based semantic retrieval with traditional keyword-based indexing.
• Vector Storage - Provides a specialized embedded storage engine for high-dimensional vectors with configurable numeric formats and similarity metrics.
• Hybrid Retrieval - Fuses vector similarity, full-text keyword matching, and scalar filtering into a single query operation.
• Vector Storage - Provides a high-performance storage engine for persisting high-dimensional dense embeddings used in semantic search.
• Hybrid Search Retrievers - Combines vector similarity with scalar filtering and weighted ranking to refine retrieval for RAG and AI agents.
• RAG Document Retrieval - Fetches relevant document snippets from a local knowledge base to provide grounded context for LLMs.
• RAG Context Retrieval - Retrieves relevant documents from a knowledge base to ground large language model responses in factual information.
• Sparse Embeddings - Generates sparse vector representations to enable efficient lexical matching and hybrid retrieval.
• Approximate Nearest Neighbor Search - Implements approximate nearest neighbor indexing to retrieve similar embeddings without scanning the entire dataset.
• Attribute Filtering - Refines vector queries using specific scalar attributes to pinpoint precise results.
• Document Upserts - Supports adding new documents to a collection or overwriting existing ones based on a unique identifier.
• Schema Definition - Defines document structures using dynamic schemas that allow field modifications without database rebuilds.
• Document Lifecycle Management - Manages the full lifecycle of documents, including insertion, updates, and deletions via unique identifiers.
• Document Deletion Operations - Provides functionality to permanently remove specific records using identifiers or conditional filters.
• Vector Document Indexing - Provides workflows for indexing single or batch documents containing dense and sparse vectors.
• Schema Modification - Allows updating scalar field names, data types, or structures after creation without downtime.
• Document Ingestion Pipelines - Implements pipelines for storing documents containing both scalar metadata and high-dimensional vector embeddings.
• Dynamic Schema Storage - Enables adding or removing scalar fields and vectors from a collection without recreating the database.
• Full Text Search - Implements full-text search capabilities for retrieving documents by matching text content through specialized indexing.
• Document Retrieval by Identifier - Enables direct retrieval of specific documents from the database using their unique identifiers.
• Hybrid Vector-Keyword Indexing - Combines dense vector embeddings with inverted keyword indices to enable fast, relevance-ranked lexical retrieval.
• Local Vector Store Backends - Manages isolated local data stores for organizing different sets of vector embeddings with local persistence.
• Embedded Persistence - Functions as a self-contained data store that persists collections locally using write-ahead logging and memory-mapped I/O.
• Local Data Persistence - Saves collections into dedicated, self-contained directories to ensure isolation and local persistence.
• Sparse - Implements a dedicated storage layer for sparse vectors to enable efficient lexical and hybrid retrieval.
• Metadata Filtering - Restricts search results using scalar field conditions to isolate specific subsets of data.
• Search and Indexing - Integrates vector index construction for similarity search and inverted scalar indexes for efficient filtering.
• Index Definitions - Defines the indexing strategy for storing document chunks and their associated metadata.
• Vector Collection Management - Provides tools for initializing and configuring vector collections with specific dimensions and scalar schemas.
• Vector Databases - Provides a specialized database engine for storing and querying high-dimensional dense and sparse vector embeddings.
• Vector Indexing - Offers a toolset for building IVF and graph-based indexes to enable approximate nearest neighbor search.
• Inverted File Indexes - Implements IVF vector indexing to partition vector spaces into clusters for faster similarity searches.
• Embedding Generation - Transforms text into high-dimensional vectors using local models or cloud APIs for semantic similarity search.
• Vector Similarity Search - Implements high-dimensional similarity search using nearest neighbor indexes to find relevant documents or images.
• Filtered Similarity Searches - Implements techniques for constraining vector similarity results using relational or full-text metadata filters.
• Knowledge Base Retrieval - Serves as a persistence layer that provides relevant document context to large language models via semantic retrieval.
• AI Agent Integrations - Exposes database interfaces and standardized protocols to connect AI agents to structured data stores.
• AI Agent Servers - Provides a standardized server interface to expose database operations as tools for AI agents.
• AI Agent Skills - Allows the definition of reusable operation sets that AI agents use to perform tasks over stored data.
• Custom Model Integrations - Allows the integration of proprietary or third-party embedding models via protocol classes for specialized representations.
• Custom Reranker Backends - Integrates external models or proprietary logic into the retrieval pipeline using a custom reranker extension system.
• Result Reranking - Provides mechanisms to re-score and re-order search hits to improve the precision of top results.
• Exclusive Write Access - Maintains exclusive write access to data collections while allowing multiple concurrent read operations.
• Billion-Scale Vector Search - Executes approximate nearest neighbor searches across billions of vectors by optimizing memory and disk usage.
• Inverted Index Engines - Utilizes inverted indexing to map terms and scalar values to document identifiers for fast keyword search.
• Scalar Attribute Filtering - Provides high-performance filtering of documents using primitive scalar attributes and optimized indices.
• Inverted Scalar Indexes - Creates inverted indices on structured scalar data fields to enable fast filtering and retrieval.
• Tokenized Text Indexes - Configures string fields for text search using customizable tokenizers for different languages.
• Concurrent Read Access - Allows multiple processes to read from the same data collection simultaneously for high-throughput retrieval.
• Index Construction - Provides utilities to create permanent indexes from staged vectors to optimize similarity search speed.
• Memory-Mapped Storage - Uses memory-mapped I/O to map files directly into the virtual address space for accelerated data retrieval.
• Range and Exact Match Filters - Accelerates retrieval using exact matches and range queries to isolate specific records.
• Search Index Management - Provides tools for configuring and managing indexes on both scalar and vector fields.
• Background Index Buffering - Implements background buffering for vector index construction to maintain retrieval speed during updates.
• Search Result Fusion Algorithms - Refines the sequence of retrieved items using models or fusion algorithms to improve relevance.
• Multimodal Search - Locates visually similar images or code snippets using natural language descriptions and semantic embeddings.
• Similarity Search - Finds the most similar items using either exact matching or approximate indexing.
• Vector Index Compression - Converts high-precision vectors into compact forms to reduce memory usage and lower query latency.
• Vector Search - Executes semantic similarity searches directly on the host device to eliminate network latency.
• Multi-Vector Fusion Search - Combines results from multiple independent vector embeddings and uses a re-ranker to fuse them.
• Similarity Thresholds - Filters retrieval results by rejecting candidates that fall outside a specified mathematical similarity distance radius.
• Text Vectorizers - Integrates embedding models to transform raw textual data into high-dimensional vector representations for similarity search.
• Write-Ahead Logging - Ensures data durability and recoverability by recording all mutations to a persistent log before applying them.
• Memory-Mapped I/O - Utilizes memory-mapped I/O to accelerate data retrieval and optimize memory usage for caching.
• Data Type Validation - Validates that every ingested field strictly conforms to the declared scalar or vector data types.
• Multi-Process Data Stores - Enables multiple processes to read from a single data collection simultaneously with exclusive write access.
• Model Adapter Plugins - Provides a plugin-based system to register third-party embedding models and rerankers through standardized adapters.
• Database Systems - Lightweight, in-process vector database.

Star history