apache/iceberg
Iceberg
Iceberg is an open table format and big data table manager designed for huge analytic datasets in cloud storage. It provides a specification for tracking large-scale datasets to maintain transactional consistency and structural integrity.
The project utilizes a standardized REST catalog interface to manage table metadata, ensuring interoperability between different compute engines. This allows diverse query engines to connect to a single table interface and maintain consistency across different processing frameworks.
Its core capabilities include managing large-scale analytic tables, coordinating concurrent data access to prevent corruption, and evolving table schemas without rewriting existing data files. It also provides mechanisms for validating REST catalog implementations to ensure specification adherence.
Features
- Big Data and Analytics - Organizes massive datasets using a high-performance table format designed for big data analytic workloads.
- Table Managers - Provides a comprehensive system for managing massive analytic datasets and coordinating concurrent read/write operations across multiple engines.
- Table Metadata Pointers - Decouples table metadata from compute engines by storing a pointer to the current metadata file in a catalog.
- Schema Evolution - Allows adding or renaming columns without rewriting underlying data files by tracking column IDs independently.
- Metadata-Driven Snapshots - Tracks table state through immutable snapshot files to provide consistent views and transactional isolation.
- Table Schemas - Updates table structures over time using a mechanism that avoids rewriting existing data files.
- Large-Scale Dataset Management - Tracks huge datasets in cloud storage to ensure transactional integrity and schema evolution for big data workloads.
- Manifest-Based File Tracking - Uses a hierarchical tree of manifest files to index data files and perform efficient partition pruning.
- Open Table Formats - Provides a high-performance open table format for huge analytic datasets in cloud storage with transactional consistency.
- Table Specifications - Implements a specification for tracking large-scale datasets to maintain structural integrity without rewriting existing data files.
- Concurrent Data Collection - Manages simultaneous read and write operations across multiple engines to prevent data corruption in shared analytic tables.
- Catalog Interfaces - Standardizes table management and discovery through a platform-independent REST API for interoperability between query engines.
- Concurrency Control - Prevents data corruption by validating that no conflicting changes occurred between the start and commit of a transaction.
- Atomic Write Coordinators - Manages simultaneous read and write operations across multiple compute engines to prevent data corruption in shared tables.
- Cross-Engine Query Interfaces - Connects diverse compute frameworks to a single table interface to maintain consistency across different processing engines.
- Catalog Specification Standards - Implements a common API specification to ensure interoperability between different catalog versions and client engines.
- Multi-Engine Interfaces - Connects diverse compute frameworks to maintain a consistent table interface regardless of the underlying processing engine.
- Catalog Interfaces - Implements a standardized REST catalog interface to manage table metadata and ensure interoperability between diverse query engines.
- Data Storage Systems - Provides high-performance, ACID-compliant table formats.
- Storage Layers - Upserts, deletes, and incremental processing.
- Data Engineering - High-performance table format for analytic datasets.
