asg017/sqlite-vec
Sqlite Vec
sqlite-vec is a C-based vector library and SQLite extension that adds virtual tables for storing and querying high-dimensional embeddings. It functions as a database plugin for performing nearest neighbor searches using distance metrics such as L2, cosine, and Hamming distance.
The project provides a portable embedding store that supports deployment across Android, iOS, desktop environments, and web browsers via WebAssembly. It distinguishes itself by converting numerical arrays into compact binary formats and utilizing quantization to reduce the memory footprint and storage size of vector indexes.
The library covers a broad range of vector operations, including similarity querying, vector arithmetic, and data transformation. It also includes capabilities for metadata filtering, key-based index sharding, and the attachment of auxiliary data to vector records.
The extension can be integrated into projects using C, C++, Go, Ruby, and Rust, and it is compatible with Datasette and distributed SQLite environments.
Features
- Vector Database Integrations - Integrates high-dimensional embedding storage and similarity search capabilities directly into SQLite.
- Vector Similarity Search - The vector search extension identifies the most similar vectors to a target query using distance metrics and optimized scans.
- Vector Storage - Transforms numerical lists into compact binary formats for efficient storage and retrieval.
- Mobile Vector Storage - The vector search extension provides pre-compiled libraries to enable high-dimensional vector search on Android and iOS hardware.
- Virtual Table Implementations - The vector search extension saves numeric vectors in virtual tables to enable efficient similarity searches.
- Semantic Search - Implements semantic search by finding the most similar data based on mathematical distance.
- SQLite Extensions - Functions as a SQLite extension that adds virtual tables for storing and querying high-dimensional embeddings.
- Native Extension Loading - Implements a mechanism to dynamically load native C search logic as an extension into the database process.
- Vector Database Extensions - Converts numerical arrays into a compact binary format optimized for integration within the database.
- Quantization - The vector search extension compresses floating point elements into smaller data types to lower the memory footprint of vector indexes.
- Vector Quantization - Maps floating-point vector elements into smaller data types to reduce the memory footprint of vector indexes.
- Vector Data Conversion - The vector search extension transforms binary blobs or text into specialized numeric formats for efficient storage.
- Virtual Tables - Implements vector data storage as a virtual table within SQLite to manage embeddings as binary blobs.
- C Libraries - Ships as a low-level C library for converting, quantizing, and storing numerical vectors.
- Distance Metrics - Computes similarity using L2, cosine, and Hamming distance metrics to rank and retrieve nearest neighbors.
- Local On-Device AI - Enables the building of AI-powered tools that perform vector search on-device or in the browser.
- Binary Serialization - Provides tools for converting high-dimensional numerical arrays into compact binary formats for database storage.
- Cross-Platform Embedding Stores - Provides a portable storage solution for vector embeddings across Android, iOS, and desktop environments.
- Extensibility Plugins - The vector search extension can be added to database projects as a dedicated plugin to enable similarity search functionality.
- Go Database Bindings - The vector search extension provides language-specific bindings and binary drivers to embed vector search capabilities within Go applications.
- Payload Storage - The vector search extension attaches large payloads to vector records for retrieval without performing separate database joins.
- Metadata Filtering - The vector search extension applies constraints to queries using indexed columns to narrow results based on specific metadata.
- Ruby Database Bindings - The vector search extension provides bindings for Ruby to store and query high-dimensional embeddings within database connections.
- Rust Database Bindings - The vector search extension allows Rust projects to link search capabilities during the build process for native database vector operations.
- Index Sharding - The vector search extension groups vectors by specific key values to restrict searches to a subset of data.
- Memory-Optimized Storage - Reduces the memory footprint and storage size of embeddings using quantization and compact binary formats.
- Normalization and Truncation - The vector search extension slices vectors to specific dimensions and normalizes the result for embedding support.
- Vector Arithmetic - The vector search extension computes the sum, difference, or mean of vectors to derive new coordinates from existing data.
- Wasm Search Modules - Provides a compiled WebAssembly module that enables vector storage and similarity search in the browser.
- WebAssembly Compilation - Compiles core search logic into WebAssembly to enable vector operations directly within web browsers.
- WebAssembly Frameworks - The vector search extension can be compiled into WebAssembly to run vector search capabilities directly within a web browser.
- Retrieval Augmented Generation - Vector search extension for SQLite databases.
- Vector Databases - Lightweight vector search extension for SQLite databases.
