L
lance-format/lance
Lance
Lance is a columnar data format and storage layer designed for high-performance random access and the persistence of multimodal data. It functions as a vector database storage system, a multimodal data store, and a versioned dataset manager.
The project distinguishes itself as a hybrid search engine that combines vector similarity search and full-text indexing on a single dataset. It provides unified storage for diverse data types including images, audio, and video, utilizing a system that lazy-loads large binary objects only when requested.
The system manages dataset evolution through schema evolution capabilities, allowing for the addition of new columns without rewriting entire tables. It tracks data state changes using transactions, branches, and tags to maintain consistent historical snapshots.
Additional capabilities include high-speed random record retrieval, secondary index mapping for rapid access, and a transaction-based write-ahead log to ensure data consistency.
Features
- Multimodal Data Storage - Provides a unified format for saving and organizing images, videos, audio, and text.
- Dataset Versioning Systems - Tracks changes to large datasets using transactions and branches to maintain a complete history.
- Columnar Storage Engines - Implements a columnar storage layout to accelerate analytical scans and reduce disk I/O.
- Schema Evolution - Allows updating data schemas by adding new columns with backfilled values without rewriting tables.
- Hybrid Search Engines - Functions as an indexing system that integrates vector-based semantic retrieval with keyword-based search.
- Hybrid Vector and Full-Text Search - Combines vector similarity search and full-text indexing on a single dataset for efficient retrieval.
- Apache Arrow-Based Formats - Provides a storage standard based on Apache Arrow for high-performance random access.
- Manifest-Based File Tracking - Uses immutable manifest files to map logical dataset versions to physical data files.
- Vector Search Indexes - Implements specialized data structures for fast similarity searches within high-dimensional embedding spaces.
- Vector Storage - Provides a persistent storage layer optimized for the retrieval and indexing of high-dimensional vectors.
- Versioned Datasets - Tracks data state changes using transactions, tags, and branches to maintain a complete history.
- Dynamic Schema Storage - Enables updating dataset columns and backfilling values without the need to rewrite the entire table.
- High-Speed Random Record Retrieval - Enables high-speed retrieval of specific records without impacting the performance of full table scans.
- Lazy-Loading Blob Storage - Provides a system that lazy-loads large binary objects separately from metadata only when requested.
- Random Access Data Retrieval - Enables high-speed retrieval of specific records without impacting the performance of full table scans.
- Secondary Indexes - Maintains separate lookup tables mapping search keys to record locations for rapid random access.
- Write-Ahead Logging - Ensures data consistency and atomicity via a transaction-based write-ahead log.
- RAG and Data Pipelines - Multimodal data format for fast vector search and access.
