lancedb/lancedb

Features

• Dataset Versioning Platforms - Automatically tracks and manages historical versions of datasets to ensure machine learning reproducibility.
• Vector Databases - Serves as a high-performance database optimized for storing and querying high-dimensional vector embeddings for semantic search.
• Vector Similarity Search - Provides core vector similarity search capabilities to find the nearest neighbors to a query vector.
• Embedding Generators - Automatically computes vector representations for data upon insertion using registered embedding functions.
• Embedding Models - Integrates with external embedding model providers to convert raw data into vector representations.
• RAG Pipelines - Coordinates document extraction, chunking, and indexing through a visual interface to build RAG workflows.
• Dataloader Integrations - Streams specific columns directly into machine learning training loops via a projection-based interface.
• Multimodal Retrieval Systems - Integrates vector similarity, full-text search, and SQL filters for precise retrieval of multimodal data.
• Text Embedding Generators - Transforms data using local models or remote APIs to generate text embeddings for semantic search.
• Training Pipelines - Integrates tables directly into ML loaders to stream observations and actions for policy training.
• Vector Embeddings - Transforms text into vectorized representations using external models for semantic search and comparison.
• Multimodal Embeddings - Creates vector representations for both text and image data to enable cross-modal semantic search.
• Derived Feature Generation - Enables the creation of new data columns from existing records using custom logic and large-table backfills.
• Analytical Query Engines - Executes high-speed analytical SQL queries by pushing operations down to the columnar storage layer.
• Approximate Nearest Neighbor Search - Implements IVF-PQ and HNSW algorithms to accelerate high-dimensional vector searches.
• Multimodal Table Management - Creates and manages tables that simultaneously store vector embeddings and scalar metadata.
• Bulk Data Ingestion - Provides high-performance utilities for importing large volumes of data using parallelized writes.
• Columnar Databases - Implements a storage engine optimized for analytical queries and efficient disk I/O using columnar data structures.
• Columnar Storage Engines - Saves contiguous observations and actions in a columnar format to enable high-performance indexed reads for research data.
• Concurrent Write Optimizations - Manages simultaneous data updates to the same table using atomic writes.
• Connection Establishment - Establishes database connections via local file paths, object storage, or remote URIs.
• Cloud Native Object Storage - Persists data across major cloud object stores like S3, GCS, and Azure Blob Storage using URI schemes.
• Schema Definition - Defines data structures, including column types and vector dimensions, using schema declarations.
• Data Ingestion - Loads multimodal data objects and automatically infers the database schema from the provided data.
• Data Ingestion Sources - Imports data from diverse sources including APIs, databases, and CSVs with automatic schema inference.
• Atomic Transaction Execution - Executes batches of table declarations and updates as single atomic transactions for consistency.
• Data Versioning - Rolls back tables to specific prior versions without duplicating data to ensure reproducibility.
• Machine Learning Data Pipelines - Streams specific columns into machine learning pipelines to avoid loading entire datasets into memory.
• Feature Engineering Tools - Provides a specialized SDK for transforming raw data into formats optimized for machine learning and vector storage.
• AI Knowledge Bases - Stores external content as embeddings and metadata to serve as a high-performance knowledge base for AI agents.
• Attribute Filtering - Combines vector search with structural filters on row attributes to narrow candidate sets.
• Scalar Attribute Filtering - Provides high-performance filtering of multimodal data using scalar predicates and bitmap indices.
• Atomic Transactions - Ensures transaction status updates are applied completely through atomic actions.
• Atomic Version Updates - Updates multiple table versions in a single batch operation to ensure atomic commits.
• Full Text Search - Provides full-text search on string columns using BM25 ranking and tokenization.
• High-Concurrency Database Access - Supports multiple simultaneous read operations and concurrent writes across a scalable storage layer.
• Hybrid Search - Merges semantic and full-text search results into a single ranked list using linear score combination.
• Hybrid Search Engines - Integrates dense vector similarity, BM25 full-text search, and scalar filters into a single ranked result set.
• Training Sample Streaming - Integrates dataset columns into data loaders to enable efficient prefetching, shuffling, and batching for models.
• Large-Scale Feature Transformations - Allows adding new columns and transforming data at scale to extend tables vertically and horizontally.
• Parallelized Ingestion - Writes massive datasets from files or iterators using parallelized batches to avoid memory exhaustion.
• Metadata Filtering - Supports complex data retrieval using boolean predicates and range filters on structured metadata.
• Multi-Backend Storage Management - Persists data across object storage, network file systems, and local disks to balance cost and latency.
• Multimodal Data Storage - Organizes AI/ML data using a specialized layout that stores observations and raw media files side-by-side.
• Object Storage Integrations - Integrates with S3, GCS, and Azure Blob Storage using URI schemes for data persistence.
• Dataset Mutation Tracking - Commits every mutation as a new version and allows pinning to tags for reproducibility.
• Search and Indexing - Implements diverse indexing types including IVF_PQ for semantic search, inverted indices for text, and B-trees.
• Vector Search Indexes - Implements IVF and HNSW algorithms to accelerate similarity searches for high-dimensional embeddings.
• Hybrid Retrieval - Combines dense vector embeddings with keyword queries to merge and rerank results for higher accuracy.
• Metadata Search Indices - Combines vector similarity search with scalar metadata filters to narrow down search results.
• Semantic Search - Retrieves semantically similar records using distance metrics and optimized vector indices.
• Multimodal Search - Integrates vector search, full-text search, and SQL filtering to retrieve relevant records across multimodal data.
• SQL Query Execution Engines - Provides a standard SQL interface for exploring and analyzing stored data.
• Storage Backend Adapters - Supports diverse storage backends including local filesystems and various cloud object storage providers.
• Structured Data Management - Provides systems for managing multimodal data with indexed schemas and structured layout.
• Table Data Processing - Executes complex table-wide transformations such as deduplication and aggregation using batch processing.
• Database Indexes - Supports the creation of both vector and scalar indices to optimize overall query performance.
• Scalar Metadata Retrieval - Allows retrieval of records based strictly on metadata criteria without requiring vector similarity.
• Vector Indexing - Provides tools for creating and managing specialized high-dimensional vector indexes like IVF-PQ and HNSW.
• Vector Memory Stores - Serves as a high-performance vector store for indexing and retrieving document embeddings in RAG pipelines.
• RAG Optimizations - Implements specialized database features optimized for high-performance indexing and retrieval within RAG pipelines.
• Vector Search - Retrieves similar vectors with tunable parameters for balancing recall, latency, and distance metrics.
• Vector Search Engines - Provides a vector search engine that combines dense vector similarity with BM25 full-text search and SQL filtering.
• Boolean-Constrained - Combines vector similarity searches with boolean predicates to restrict results to specific subsets.
• Ingestion-Time Embedding Generation - Computes dense or multi-vector embeddings automatically during insertion via an integrated inference engine.
• Filtered Similarity Searches - Narrows vector similarity results by applying metadata filters to target specific categories.
• Versioned Storage - Restores tables to historical versions to reproduce training runs or audit states.
• User-Defined Data Functions - Provides the ability to define custom functions for computing new data features or bucket values based on column data.
• Version Pinning Tools - Implements systems for locking datasets to specific versions to ensure consistency and reproducibility.
• Dataset State Pinning - Allows users to pin specific dataset states via tags to ensure stability across experiments.
• Table - Creates new version entries for tables to enable tracking of changes and snapshots over time.
• Dataset Version Snapshots - Records every mutation as a discrete version with named tags for reproducible AI experiments and data time-travel.
• Synthetic Dataset Generators - Converts source documents into question-answer pairs and summarizations to create synthetic fine-tuning datasets.
• Dataset Preparation Tools - Creates custom permutations of datasets through filtering and shuffling to prevent model overfitting.
• AI-Powered Dataset Analyzers - Uses large language models as judges to filter high-quality examples from generated datasets.
• Embedding Generation - Automatically generates vector embeddings for specific fields during the data ingestion process.
• Experiment Tracking - Tracks every mutation as a distinct version and supports tagging for machine learning experiment tracking.
• Feature Extraction Models - Extracts features from raw multimodal data for AI models using scalable local and distributed functions.
• Multi-Value - Produces several related features, such as dimensions and embeddings, from a single operation.
• Fine-Tuning Data Exporters - Saves curated data into formats compatible with training workflows like JSONL and JSON.
• High-Dimensional Data Loading - Retrieves temporal windows of high-dimensional data from storage to maximize GPU utilization during training.
• Data Loaders - Provides a projection layer that plugs into standard PyTorch data loaders for streaming model training data.
• Local Embedding Generators - Transforms text into vector embeddings using local worker models to eliminate external API costs.
• Model-Driven Text Extraction - Provides multimodal model-driven text extraction from documents, images, audio, and video files.
• Model Comparison Interfaces - Provides an analytical approach to comparing outputs from multiple models or prompts stored in parallel columns.
• Retrieval Strategy Evaluation - Allows comparison of embedding models, chunking strategies, and indices to identify the most performant search setup.
• Multimodal Embedding Models - Uses CLIP models to convert images and text into a shared vector space for cross-modal retrieval.
• Result Reranking - Provides an extensible framework for implementing custom reranking algorithms to refine search result precision.
• Embedding Model Registries - Provides a registry to map source data columns to specific vector embedding models.
• OpenAI Model Integrations - Converts text data into vectors using OpenAI models for semantic and multimodal retrieval.
• Sentence Embeddings - Converts text into normalized sentence-level vectors using Sentence Transformers with hardware acceleration.
• Late Interaction Retrieval - Implements late-interaction techniques by storing multiple embeddings per item for high-precision semantic retrieval.
• Cross-Modal Retrieval Frameworks - Enables searching across different media types by leveraging unified embeddings for cross-modal retrieval.
• Document Chunking Expansion - Provides a mechanism to split single source rows into multiple child rows, essential for RAG pipelines.
• Embedding Model Integrations - Transforms text into vectors using models sourced from the Hugging Face Hub.
• Data Integration Tools - Syncs and exchanges data with external data engineering tools and columnar processing engines.
• Dataset Curation and Generation - Slices, samples, and filters large multimodal datasets to prepare high-quality data for model training.
• Data-Code Version Linking - Links specific model checkpoints to the exact version of the data used during training.
• GPU-Accelerated Vector Indexing - Uses GPU hardware to build IVF and HNSW indexes to reduce indexing time for large datasets.
• Asynchronous Database Drivers - Provides non-blocking asynchronous calls for database connections, table creation, and searches.
• Batch Ingestion Streaming - Consumes data from iterators in batches to maintain a bounded memory footprint during ingestion.
• Boolean Query Languages - Supports complex filtering expressions using boolean logic to find documents satisfying multiple conditions.
• Categorical Index Management - Implements fast categorical filtering using B-tree and bitmap indices on identifiers.
• Column Transformation - Provides capabilities to add new computed columns to datasets using SQL expressions or table merges.
• Training Data Projections - Reads a specific subset of columns during a training loop to reduce memory overhead.
• Column Value Extraction - Creates new data columns by transforming existing values through SQL expressions or external data merges.
• Conditional Data Filters - Extracts specific subsets of data using SQL predicates to filter rows based on scalar values.
• Consistency Tuning - Balances data freshness and query latency by choosing between strong consistency or defined update intervals.
• File Ingestion Services - Imports content from PDF, HTML, and DOCX files into columnar storage for efficient processing.
• Storage Compaction Utilities - Merges small data fragments into larger ones to optimize read performance and reclaim disk space.
• Column Definitions - Updates existing column definitions by renaming fields or altering data types.
• Virtual Column Functions - Defines user-defined functions to compute virtual columns that recompute when source data changes.
• Schema Evolution - Manages updates to existing data structures through column appending and external data merging.
• Schema Mapping - Automatically converts Pydantic model definitions into database schemas for table creation and validation.
• Columnar Projection Streaming - Reduces I/O overhead by selecting only specific columns and batches of data for processing.
• Table Version Restoration - Reverts a table to a previous state by restoring it to a specific historical version.
• Version History Retrieval - Lists all available commits of a table along with their metadata to track changes over time.
• Version Tagging - Assigns named labels to specific versions of a table to provide stable references.
• Table Lifecycle Automators - Manages the full lifecycle of tables, including creation, opening, and permanent deletion.
• PyTorch Streaming Integrations - Projects specific columns into formats compatible with standard PyTorch data loaders for efficient batching.
• Data Upsert Operations - Updates existing rows based on matching columns and inserts new rows to synchronize datasets.
• Table Schemas - Extends existing table schemas by adding new columns using SQL expressions or default values.
• Schema Modification - Enables updates to table structures by renaming columns or modifying data type constraints.
• Table Creation - Enables building storage tables from JSON records, DataFrames, or predefined schemas.
• Full-Text Search Indexes - Creates searchable full-text indices on text columns with customizable tokenizers and stemming.
• Dataframe Engines - Ingests Pandas DataFrames directly into tables to bridge vector storage and data analysis workflows.
• Indexed Predicate Filtering - Narrows results using SQL-like expressions on indexed columns to retrieve specific data subsets.
• External Data Integrations - Integrates external tables or arrays into existing datasets using common identifiers to enrich data.
• External Column Merges - Adds new columns to existing tables by joining with external data sources or using SQL expressions.
• Federated Databases - Sets up namespace-backed database federation using directory or REST implementations.
• Predicate-Based Metadata Curation - Allows extraction of specific data subsets through SQL-like predicates executed over metadata columns.
• Generated Columns - Runs asynchronous processes to populate virtual, function-backed columns across existing tables.
• High Availability Architectures - Uses component-level replication and load balancing to prevent downtime during node failures.
• Stateless Compute Scaling - Decouples query serving from storage and background processing to scale read traffic and indexing independently.
• Horizontal Scaling - Provides the ability to distribute query traffic and search execution across multiple nodes to scale with data volume.
• Hybrid Vector-Keyword Indexing - Builds searchable indices on text columns to enable efficient keyword-based retrieval.
• Keyword and Phrase Indexing - Indexes text columns to support keyword and phrase-based queries with configurable stop-word removal.
• Incremental Indexing Engines - Adds newly ingested data to existing indices incrementally to avoid costly full re-indexing.
• Indexing and Search - Provides BTree and Bitmap indices on non-vector columns to accelerate filtered queries and joins.
• Key-Based Merging - Synchronizes incoming datasets with existing tables by matching keys to insert or update records.
• ML Dataset Lazy-Loading - Loads data lazily from remote stores and projects specific columns to minimize network transfer.
• Materialized Views - Creates persisted views that transform source data and store derived embeddings for high-performance search.
• Custom Transformation Views - Executes user-defined logic to enrich rows or aggregate datasets into new schemas via materialized views.
• Filtered - Defines named SQL filters as views that synchronize with a source table and track version history.
• Incremental View Refreshes - Updates persisted views incrementally by processing only new or changed rows from the source.
• Persistent Transformation Views - Creates views that materialize batch functions and refreshes them when source data versions change.
• Refresh - Starts asynchronous background jobs to update materialized views using a tracking ID.
• Incremental View Refreshes - Provides the ability to update expanded views by processing only new or updated source rows.
• UDF-Based Row Enrichments - Adds computed columns to a materialized view using user-defined functions and backfills existing rows.
• Multivector Search - Supports late interaction search by matching query vectors against documents containing multiple embeddings.
• Namespace Resolution - Resolves table locations and credentials via namespace-scoped identifiers using a remote catalog.
• Attribute-Based Subset Retrieval - Retrieves data subsets using SQL-like predicates on indexed columns to isolate specific categories.
• Predicate-Based Subset Isolation - Executes scans using SQL-like predicates to isolate specific data subsets based on attributes.
• Phrase Sequence Matching - Enables precise searching for specific word sequences with optional slop parameters.
• Predicate Pushdown - Executes SQL-like filters directly at the storage layer to reduce data transfer during queries.
• Query Performance Analyzers - Inspects execution plans to verify index usage and filter pushdown for improved query performance.
• Query Performance Monitors - Returns detailed execution statistics for processed queries to identify performance bottlenecks and optimize efficiency.
• Query Planning - Reveals the logical structure of a query before execution to verify optimization and filter pushdown strategies.
• Result Streaming - Iterates through table data in batches via cursors to handle large datasets without memory exhaustion.
• Remote Table Connectivity - Connects to storage-backed tables over a network using logical identifiers, removing the need for local paths.
• Row Expansion - Splits single source rows into multiple rows to decompose complex documents or media into smaller chunks.
• Schema Evolutions - LanceDB adds new data fields to an existing table via SQL expressions or external table merges.
• Schema Extensions - Extends base database schemas by adding new columns via SQL expressions or external label merges.
• Search Indexing - Creates BTree or Bitmap indices on numeric or categorical columns to accelerate metadata filtering.
• BM25 Full-Text Indices - Implements BM25-based full-text search indices on string columns to enable efficient keyword retrieval.
• Incremental Index Integration - Integrates newly added rows into existing indices to maintain search performance and completeness.
• Index Lifecycle Management - Automatically handles index creation and optimization in the background as data changes.
• Automatic Background Indexing - Triggers index updates automatically as new data is added to minimize latency impact.
• Search Result Filtering - Provides tools to filter, boost, and combine keyword matches using fuzzy search and boolean logic.
• Secondary Indexes - Combines vector and full-text search with secondary indexes to accelerate non-primary key lookups.
• Similarity Search - Enables similarity search across different media types, allowing queries using either text or images.
• Storage Space Optimization - Permanently removes soft-deleted rows and optimizes data files to reclaim storage space.
• Structured Data Extraction - Identifies and extracts specific entities from text columns to create structured data for filtering.
• Column Deletions - Removes specific columns from a table by identifying and dropping the corresponding field paths.
• Asynchronous Indexing - Builds vector or scalar indexes on table fields asynchronously to accelerate search operations.
• Text Pattern Matching - Locates records by matching specific text strings and patterns across the entire dataset.
• Computed Logic Recalculation - Allows revising the logic of a computed column and triggering a recalculation of table values.
• GPU-Accelerated Indexing - Leverages GPU hardware to index billions of vectors significantly faster than CPU-only processing.
• Vector Quantization - Implements vector quantization to compress high-dimensional embeddings and reduce storage requirements.
• Remote - Performs vector and full-text searches on remote datasets via URIs without requiring local downloads.
• Custom Embedding Logic - Supports proprietary embedding logic by allowing extensions of base classes for specialized data types.
• Similarity Thresholds - Allows filtering search results to only include vectors that fall within specific similarity thresholds.
• Column Projection - Reads only the required columns for a task to reduce memory overhead and disk I/O.
• Projected Column Streaming - Reads only required columns from a table into a loop to minimize memory usage and disk I/O.
• Zero-Copy Data Access - Implements random-access indexing and zero-copy reads to feed data batches directly into training loops without overhead.
• Zero-Copy Data Ingestion - Loads data from Polars DataFrames using zero-copy transfers for maximum ingestion performance.
• Multi-Language SDKs - Provides native and wrapper libraries across multiple programming languages to manage and query vector data.
• REST APIs - Exposes a standardized OpenAPI protocol for reading, writing, and managing tables through a metadata service.
• Catalog Interfaces - Uses a REST-based implementation to resolve table locations and manage metadata across remote object stores.
• Dataset Version Tagging - Assigns, updates, and deletes descriptive labels for table versions to simplify identifying data snapshots.
• Evaluation State Tagging - Enables tagging of specific data mutations to mark stable states for model evaluation and training.
• Background Optimization Processes - Runs file compaction and table optimization as asynchronous background tasks to maintain system performance.
• Customer-Owned Cloud Installations - Installs the database into a user-owned cloud account to satisfy strict data residency requirements.
• Distributed Feature Engineering Deployments - Runs distributed data transformation workflows across Kubernetes clusters using Ray for large-scale datasets.
• Array Processing - Improves performance by processing multiple rows simultaneously using array-based batching instead of individual values.
• Object Store Encryption - Protects stored information in the object store and cache using encryption at rest.
• Data Isolation Strategies - Maintains strict data isolation protocols to ensure information remains confined to the owner's account.
• Data Residency Controls - Controls where infrastructure and storage reside by deploying the database into customer-owned accounts.
• Asynchronous Background Processors - Offloads index construction and data compaction to background worker threads to maintain query performance.
• Prompt Configuration Testing - Enables the evaluation of different prompts and configurations to determine their effectiveness with large language models.
• Predicate-Based Filtering - Provides capabilities to filter datasets using boolean evaluator functions and content predicates.
• Database Systems - Serverless, low-latency vector database for AI applications.

Star history