apache/hudi
Hudi
Apache Hudi is an open-source table format that brings ACID transactions, incremental processing, and multi-modal indexing to data lakes. It provides atomic commits with snapshot isolation, rollback, and optimistic concurrency control for reliable data lake operations, while supporting upserts, record-level updates, and deletions in large analytical datasets.
The project distinguishes itself through a timeline-based architecture that coordinates all write operations, enabling features like time-travel querying, incremental change streaming, and multi-modal query views that include snapshot, incremental, and read-optimized modes. It maintains pluggable indexes for fast upserts and queries, and uses a hybrid row-columnar storage approach that merges row-based and columnar formats asynchronously for balanced write and read performance.
Hudi supports both batch and streaming data ingestion from sources like Apache Spark, Apache Flink, and Kafka, with built-in CDC capabilities. It automates table maintenance services including compaction, cleaning, clustering, and indexing, and provides schema evolution with enforcement at write time. The project also offers savepoints for data recovery and incremental data processing capabilities that replace batch pipelines with streaming on data lakes.
Features
- Transactional Data Lake Engines - Guarantees atomic commits, rollback, and snapshot isolation for reliable data lake operations.
- ACID Transactional Cores - Provides atomic writes, snapshot isolation, and non-blocking concurrency controls for reliable data lake transactions.
- Data Lake Transaction Managers - Commits data atomically with rollback, snapshot isolation, and optimistic concurrency control for consistent writes on data lakes.
- Timeline-Coordinated Commits - Coordinates all write operations through a central timeline that records commits, rollbacks, and savepoints for atomicity and recovery.
- Hybrid Storage Formats - Stores data in a mix of row-based (Avro) and columnar (Parquet) formats, merging them asynchronously for balanced performance.
- Multi-Modal Query Views - Supports snapshot, incremental, time-travel, and read-optimized query modes by combining timeline metadata with storage formats.
- Batch Table Ingesters - Writes batch data into a table using Spark DataSource API, supporting upsert and insert modes.
- Record-Level Indexes - Maintains scalable record-level and column-level indexes automatically during writes to speed up lookups.
- Transactional Data Lake Storage - Brings ACID transactions and incremental processing to data lakes built on Apache Spark and Apache Flink.
- Table Performance Optimizers - Continuously schedules and orchestrates clustering, compaction, cleaning, file sizing, and indexing to maintain high performance.
- Data Upsert Operations - Inserts new records and updates existing ones in large datasets using fast, pluggable indexing for efficient lookups.
- Schema Evolution - Evolves table schemas over time while enforcing schema compliance at write time for pipeline resilience.
- Multi-Modal Indexing Subsystems - Maintains record-level and column-level indexes to accelerate upserts and queries on large tables.
- Real-Time Data Aggregators - Provides real-time snapshot querying by combining row-based and columnar storage for consistent views.
- Record Deletion - Supports record-level deletions from tables using conditional predicates via the Spark DataSource API.
- Pluggable Indexing Engines - Uses configurable indexes (Bloom filter, HBase, bucket) to map record keys to file locations for fast upserts and queries.
- Open Table Formats - Implements an open-source table format that manages atomic writes, snapshot isolation, and upserts on cloud storage and HDFS.
- Stream Ingestion - Ingests both streaming and batch data from Spark, Flink, and Kafka into a transactional data lake.
- Index-Accelerated Querying - Speeds up write transactions and query performance on large tables using a multimodal indexing subsystem.
- Table Format Integrations - Reads data from Hudi tables using standard Spark SQL or DataFrame APIs for analytical processing.
- Incremental Data Pipelines - Replaces batch pipelines with incremental streaming on data lakes using commit-based change feeds.
- Table Maintenance Automations - Runs cleaning, clustering, compaction, and indexing services automatically with configurable scheduling and failure handling.
- Table Maintenance Services - Provides automated background services for compaction, cleaning, and indexing to maintain table health.
- File Group Organizers - Organizes data into file groups that are periodically clustered to optimize file sizes and query efficiency.
- Database Atomic Batches - Commits batches of changes as a single atomic unit and rolls back on failure to keep the dataset consistent.
- Time-Travel Query Interfaces - Rolls back to table versions, debugs data changes over time, and audits commit history for historical analysis.
- Multi-View Query Interfaces - Queries tables as snapshot, incremental, change-data-capture, time-travel, or read-optimized views using indexes.
- Hybrid Storage Compactors - Asynchronously merges row-based and columnar storage to maintain query performance and manage file sizes.
- Columnar Snapshot Optimizers - Serves snapshot queries using only columnar storage for high performance on analytical workloads.
- Write-Time Schema Validators - Validates and evolves table schemas at write time, failing fast on mismatches to ensure pipeline consistency and data integrity.
- Table Change Streams - Delivers a feed of records inserted or updated after a given point in time for downstream processing.
- Incremental Change Notifications - Reads only the rows that changed since a given commit or time to support incremental data pipelines.
- Data Storage Systems - Adds transactional capabilities to data lakes.
- Storage Layers - Upserts, deletes, and incremental processing.
- Data Engineering - Open data lakehouse platform for high-performance storage.
