eto-ai/lance

Features

• Hybrid Search Methods - Integrates vector similarity search, full-text search, and SQL relational filters into a single hybrid retrieval system.
• Lakehouse Table Formats - Implements a storage format for large-scale AI datasets that brings ACID transactions and high-performance random access to multimodal data.
• Multimodal Data Storage - Organizes large-scale multimodal datasets into a file and table format optimized for high-performance vector search.
• Image Tensor Storage - Manages images as multi-dimensional numeric arrays for direct use in machine learning models and tensor operations.
• Columnar-Blob Hybrid Storage - Combines a columnar layout for structured data with a specialized blob store for large multimodal tensors.
• Columnar Storage Engines - Provides a storage layout organized by column with ACID transactions for efficient analytical workloads and versioning.
• Dataset Creation - Writes data from tables, dataframes, or streams into a high-performance columnar format.
• S3-Compatible Cloud Storage - Integrates with S3-compatible object storage providers for persisting large-scale datasets and multimodal assets.
• Schema Evolution - Updates table structures by modifying manifest metadata to add or remove columns without rewriting data files.
• ACID Dataset Versioning - Tracks dataset changes via ACID transactions, supporting time travel, branching, and tagging for reproducible AI experiments.
• Blob Dataset Management - Provides a management layer for S3-compatible storage that handles large binary blobs and tensors with lazy loading.
• Hybrid Search Engines - Combines vector-based semantic retrieval, traditional keyword indexing, and SQL analytics in one system.
• Lazy-Loading Blob Storage - Stores multimedia content using a specialized type that supports lazy loading to prevent memory overload.
• Manifest-Based Versioning - Tracks dataset state via immutable manifests to enable ACID transactions, time travel, and dataset branching.
• Object-Storage Persistence - Persists datasets across cloud providers and distributed filesystems using URI schemes to specify storage backends.
• Random Access Data Retrieval - Provides high-speed retrieval of specific records from massive columnar datasets without requiring sequential scans.
• Schema Evolutions - Adds new columns with backfilled values to existing tables without requiring a full dataset rewrite.
• ACID-Compliant - Tracks changes via automatic versioning and ACID transactions to support time travel, tagging, and branching.
• Vector Databases - Indexes and stores high-dimensional embeddings to enable fast similarity search across large datasets.
• Inverted File Indexes - Uses inverted file indexes with product quantization to enable fast approximate nearest neighbor search on embeddings.
• Vector Similarity Search - Implements high-performance nearest neighbor search for vector embeddings using IVF-PQ indexing.
• Vector Storage - Provides optimized storage for high-dimensional numerical embeddings to facilitate SIMD-accelerated distance computations.
• Column Projection - Retrieves specific columns and rows using projection and SQL predicate push-down to minimize I/O.
• Streaming Dataset Loaders - Reads large datasets in incremental batches to process data that exceeds available system memory.
• Random-Access Dataset Streaming - Implements storage-level random access to rows by index to optimize sampling and shuffling for machine learning training.
• JSON Array Functions - Extracts values, checks existence, or measures array lengths within JSON columns using JSONPath syntax.
• Binary Blob Retrieval - Fetches binary objects using positional indices, logical row IDs, or physical addresses to locate specific data.
• Column Renamers - Implements utilities for changing the names of top-level or nested columns within a dataset.
• Nested Columnar Storage - Saves nested records and dynamic key-value maps within single columns of the columnar format.
• Data File Compaction - Rewrites small data fragments into larger files and removes deleted rows to improve storage layout and scan performance.
• Table Metadata Updates - Updates table structures and backfills data via metadata modifications to avoid costly full-dataset rewrites.
• Data Value Updates - Modifies existing column values using a left-outer-hash-join between dataset fragments and update data.
• In-Memory Caching - Maintains vector and scalar indices in a memory cache to improve query response times and minimize disk I/O.
• Roaring Bitmaps - Implements compressed inverted lookup tables using roaring bitmaps to accelerate filtering on low-cardinality columns.
• Sorted Indexing - Provides BTree-based sorted indexing to facilitate efficient range queries and ordered data retrieval.
• Incremental Data Appending - Inserts new records into existing tables to grow datasets via bulk writes or append modes.
• Index Construction - Constructs specialized indices to enable high-performance similarity search across large-scale multimodal datasets.
• Parallel Construction - Coordinates multiple workers to build index segments in parallel, optimizing the construction process across CPU cores.
• Hybrid Relational-JSON Storage - Encodes JSON as binary JSONB for efficient storage and fast query performance for nested fields.
• Full-Text JSON Indexes - Builds inverted indices over JSON documents to enable comprehensive text-based search across all contained content.
• Metadata Caching - Caches manifests, transactions, and file metadata in memory to accelerate data access and reduce repeated lookups.
• Optimistic Concurrency Control - Manages simultaneous writes by validating manifest versions before commit to ensure data consistency without locking.
• Parallel Storage Writing - Generates data fragments in parallel across multiple nodes and commits them into a single dataset.
• Row Deletions - Removes specific records from a dataset using SQL expressions to mark rows as deleted.
• Search Index Management - Treats search indices as versioned table objects decoupled from file encoding to support independent evolution.
• SQL-Based Row Updates - Modifies existing record values using SQL expressions to apply changes to specific rows.
• Binary Data Streaming - Retrieves large objects as lazy handles to stream bytes on demand without loading the entire object.
• Column Deletions - Provides the ability to remove individual columns from a table schema via metadata updates.
• Bitmap Indexes - Uses Roaring Bitmaps to generate compressed inverted lookup tables for high-performance filtering on low-cardinality data.
• Upsert Operations - Adds new data in bulk while matching against existing records to handle updates and insertions in one operation.
• Dataset Version Tagging - Assigns named tags to specific dataset versions to track evolution and protect important snapshots.
• Dataset Branching - Creates independent branches from specific versions or tags to allow separate lines of development and writes.
• BTree Range Indexes - Implements a two-level sorted BTree structure to enable efficient range queries and ordered data retrieval.
• Data Management - Implements a modern columnar data format for ML.

Star history