delta-io/delta

Features

• Lakehouse Storage Layers - Unifies data lake scalability with data warehouse consistency to create a high-performance storage layer.
• Lakehouse Table Formats - Implements a lakehouse table format that ensures transactional consistency across diverse compute engines.
• Transaction Management - Implements a transaction log to ensure consistency and serializable isolation across multiple compute engines.
• Transaction-Log-Based Versioning - Tracks every table modification in a sequential log to enable ACID transactions and point-in-time snapshots.
• ACID Transaction Semantics - Guarantees data consistency through serializable isolation to prevent corruption during concurrent operations.
• ACID Transactional Cores - Coordinates atomic commits and serializable isolation for concurrent reads and writes across distributed compute engines.
• Catalog-Based Commit Coordinators - Provides a centralized catalog service to coordinate atomic commits across distributed compute clusters.
• Atomic Table Mutations - Performs deletes, updates, and merges on data tables to maintain accurate records over time.
• Change Data Capture - Tracks row-level changes in a table to serve as a source for downstream incremental processing.
• Change Data Capture Systems - Exposes table updates and deletes as a streaming source for incremental data pipelines.
• Compute Engine Integrations - Links SQL engines, data processing frameworks, and databases to read from or write to existing tables.
• Concurrency Control - Implements optimistic concurrency control to detect conflicting writes and maintain data consistency without locking.
• Write Skew Mitigations - Prevents data corruption and write skew anomalies by detecting conflicts between simultaneous write operations.
• Historical Data Querying Interfaces - Provides interfaces to retrieve past table states and historical data using the transaction log.
• Time-Travel Query Interfaces - Enables time-travel queries to retrieve table states using specific snapshot identifiers or timestamps.
• Cloud Native Object Storage - Manages large scale data tables across cloud object stores with metadata-driven data skipping.
• Metadata-Driven Data Skipping - Scans metadata files using predicates to identify specific data partitions or files to optimize query performance.
• Data Format Interoperability - Translates data between different lakehouse formats to ensure compatibility across ecosystem standards.
• Data Versioning - Tracks historical table versions to enable rollbacks, audit trails, and reproducible experiments.
• Data Versioning - Maintains historical snapshots of data to enable time-travel analysis and state reproduction.
• Exactly-Once Processing Semantics - Guarantees exactly-once semantics during ingestion using a coordinator and committer to prevent data duplication or loss.
• Data Normalization and Schema Enforcement - Validates that incoming data matches the defined table structure to prevent corruption.
• Atomic Transactions - Enforces a serial order for concurrent writers to ensure atomic transactions and consistent snapshots.
• Atomic Transactional Commits - Brokers table writes through a centralized catalog to coordinate atomic transactional commits.
• Atomic Version Updates - Writes metadata changes to a transaction log as a single indivisible commit to ensure consistency.
• Table Schemas - Enforces and evolves table structures over time to ensure data consistency.
• Schema Evolution - Tracks structural changes in a transaction log to map physical files to logical views without full rewrites.
• Table Metadata Updates - Tracks structural and administrative table metadata updates via a transaction log.
• Database Record Operations - Implements standard DML operations including updates and deletes on data records through a consistent API.
• Cloud Object Storage - Manages structured data tables across cloud object stores while ensuring transactional guarantees.
• Insert Operations - Provides capabilities for adding new records to tables via atomic commits.
• Metadata Scaling - Organizes billions of files and partitions across petabyte-scale tables to maintain high retrieval performance.
• Multi-Engine Table Interfaces - Provides a consistent set of APIs for interacting with data tables across different compute engines and storage systems.
• Object Storage Integrations - Manages large-scale data tables across S3, Azure, and GCS while maintaining transactional guarantees.
• Record Updating - Performs atomic deletes, updates, and merges to modify existing records without full dataset rewrites.
• Open Table Formats - Generates compatible metadata to enable interoperability with open table formats like Iceberg and Hudi.
• Strict Serializability - Ensures a strict serial order for concurrent writes to prevent corruption and provide consistent views for readers.
• Atomic Table Mutations - Updates, deletes, or merges records into existing tables using batch jobs or programmatic APIs.
• Table Data Retrieval - Retrieves records from tables by specifying the physical file paths where the data is stored.
• Transactional Data Mutations - Provides transactional persistence for delete, update, and merge operations using a transaction log to maintain consistency.
• Multi-Engine Interfaces - Employs a consistent kernel and set of APIs to allow diverse query engines to read and write open table formats.
• Stream-Table Unification - Unifies real-time event streams with structured table formats to read continuous sequences of updates.
• Change Tracking - Maintains a detailed transaction log of all table modifications to provide a complete audit trail.
• Change Data Capture - Exposes row-level modifications as an event stream to facilitate incremental data pipelines.
• Cross-Format Metadata Synchronization - Generates compatible metadata for external table formats at commit time to enable cross-engine interoperability.
• Table Change Streams - Consumes tables as streaming sources to process changes starting from specific versions or timestamps.
• Data Validation Schemas - Enforces defined table structures during data ingestion to prevent corruption and manage schema evolution.
• Concurrent Write Coordinators - Arbitrates simultaneous write attempts via a centralized catalog to prevent data loss.
• Catalog-Based Coordination - Shifts transaction coordination from the filesystem to a central catalog to establish a single source of truth.
• External Catalog Coordination - Coordinates table creation, reading, and writing by shifting commit management to a centralized catalog.
• Atomic Table Swaps - Performs table replacements and partition overwrites as single atomic commits.
• Cloud Storage Integrations - Connects to remote cloud storage systems like S3, Azure, and GCS to perform multi-cluster reads and writes.
• Compute Engine Connectors - Integrates shared tables directly into various compute engines and analysis tools regardless of the deployment pattern.
• Multi-Cluster Write Coordination - Coordinates concurrent write operations from multiple compute clusters using external locks.
• Data Ingestion - Provides mechanisms to ingest real-time data streams from sources like Kafka into lakehouse tables.
• Table Format Translators - Generates compatible metadata for external open table formats to allow cross-engine interoperability without data duplication.
• Custom Connector Development - Provides low-level protocol interfaces for developing custom connectors that read and write massive datasets across distributed engines.
• Metadata-Driven Data Skipping - Skips irrelevant data files using partition info and file statistics to reduce I/O overhead.
• Kafka Connectors - Implements connectors to stream data from Kafka topics directly into lakehouse tables.
• Unified Batch and Stream Processing Engines - Unifies the processing of historical backfills and real-time streams using exactly-once semantics.
• Semi-Structured Data Integration - Stores flexible data in single columns and decomposes frequently accessed fields for faster reads.
• Data Upsert Operations - Supports change-data-capture and slowly-changing-dimension workflows through combined insertion and update operations.
• Data Warehousing Patterns - Implements complex warehouse architectures using techniques such as slowly changing dimensions.
• Atomic Table Creations - Guarantees that table creation from queries completes fully or not at all to prevent corrupted data.
• Storage Sharing - Distributes data securely across multiple cloud providers and organizational accounts using standardized protocols.
• Dynamic Clustering Layouts - Dynamically organizes data distribution based on column values to optimize query performance without relying on static partitioning.
• External Data Writing - Connects data processing frameworks and streaming services to write data directly into tables.
• Metadata Caching - Retrieves commit history and metadata from a catalog to bypass slow cloud storage lookups.
• Parquet Data Processing - Performs querying and analysis of Parquet files directly from storage based on specific table snapshots.
• Data Layout Optimizers - Applies advanced sorting and data skipping techniques to reduce the volume of scanned data.
• Metadata-Driven Optimizers - Leverages table-level metadata, such as file statistics and partition predicates, to skip irrelevant data during query execution.
• Query Performance Tuning - Accelerates data retrieval through liquid clustering and metadata-driven data skipping.
• Real-Time Data Streaming - Integrates real-time data streams into tables using specialized connectors for streaming engines.
• Database Constraints - Enforces database-level constraints at the catalog level to maintain data integrity.
• Secure Data Sharing - Exposes live datasets to external recipients using an open protocol to eliminate the need for data duplication.
• Slowly Changing Dimensions - Tracks historical changes to dimension data over time to enable point-in-time reporting and analysis.
• Catalog-Based Table Resolution - Eliminates filesystem path dependencies by resolving logical table identifiers via a central catalog.
• Table Initializations - Writes DataFrames to storage using specialized formats to initialize tables based on inferred schemas.
• Stream Ingestion - Writes continuous record flows to tables with exactly-once processing guarantees during concurrent operations.
• Unified Data Connector Interfaces - Provides a consistent kernel and connector set for diverse engines to read and write data using a single interface.
• External Table Querying - Connects compute engines, databases, and query tools to read data from tables using a consistent set of APIs.
• Data Management - Storage layer for ACID transactions on data lakes.
• Data Storage Systems - Brings ACID transactions to big-data storage layers.
• Storage Layers - Storage layer with ACID transactions.
• Data Engineering - Storage layer bringing ACID transactions to big data.

Star history