awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
apache avatar

apache/iceberg

0
View on GitHub↗
iceberg.apache.org↗

Iceberg

Iceberg is an open table format and big data table manager designed for huge analytic datasets in cloud storage. It provides a specification for tracking large-scale datasets to maintain transactional consistency and structural integrity.

The project utilizes a standardized REST catalog interface to manage table metadata, ensuring interoperability between different compute engines. This allows diverse query engines to connect to a single table interface and maintain consistency across different processing frameworks.

Its core capabilities include managing large-scale analytic tables, coordinating concurrent data access to prevent corruption, and evolving table schemas without rewriting existing data files. It also provides mechanisms for validating REST catalog implementations to ensure specification adherence.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

Features

  • Big Data and Analytics - Organizes massive datasets using a high-performance table format designed for big data analytic workloads.
  • Table Managers - Provides a comprehensive system for managing massive analytic datasets and coordinating concurrent read/write operations across multiple engines.
  • Table Metadata Pointers - Decouples table metadata from compute engines by storing a pointer to the current metadata file in a catalog.
  • Schema Evolution - Allows adding or renaming columns without rewriting underlying data files by tracking column IDs independently.
  • Metadata-Driven Snapshots - Tracks table state through immutable snapshot files to provide consistent views and transactional isolation.
  • Table Schemas - Updates table structures over time using a mechanism that avoids rewriting existing data files.
  • Large-Scale Dataset Management - Tracks huge datasets in cloud storage to ensure transactional integrity and schema evolution for big data workloads.
  • Manifest-Based File Tracking - Uses a hierarchical tree of manifest files to index data files and perform efficient partition pruning.
  • Open Table Formats - Provides a high-performance open table format for huge analytic datasets in cloud storage with transactional consistency.
  • Table Specifications - Implements a specification for tracking large-scale datasets to maintain structural integrity without rewriting existing data files.
  • Concurrent Data Collection - Manages simultaneous read and write operations across multiple engines to prevent data corruption in shared analytic tables.
  • Catalog Interfaces - Standardizes table management and discovery through a platform-independent REST API for interoperability between query engines.
  • Concurrency Control - Prevents data corruption by validating that no conflicting changes occurred between the start and commit of a transaction.
  • Atomic Write Coordinators - Manages simultaneous read and write operations across multiple compute engines to prevent data corruption in shared tables.
  • Cross-Engine Query Interfaces - Connects diverse compute frameworks to a single table interface to maintain consistency across different processing engines.
  • Catalog Specification Standards - Implements a common API specification to ensure interoperability between different catalog versions and client engines.
  • Multi-Engine Interfaces - Connects diverse compute frameworks to maintain a consistent table interface regardless of the underlying processing engine.
  • Catalog Interfaces - Implements a standardized REST catalog interface to manage table metadata and ensure interoperability between diverse query engines.
  • Data Storage Systems - Provides high-performance, ACID-compliant table formats.
  • Storage Layers - Upserts, deletes, and incremental processing.
  • Data Engineering - High-performance table format for analytic datasets.
8,972 stars·3,318 forks·Java·Apache-2.0·20 views

Star history

Star history chart for apache/icebergStar history chart for apache/iceberg

How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.

Frequently asked questions

What does apache/iceberg do?

Iceberg is an open table format and big data table manager designed for huge analytic datasets in cloud storage. It provides a specification for tracking large-scale datasets to maintain transactional consistency and structural integrity.

What are the main features of apache/iceberg?

The main features of apache/iceberg are: Big Data and Analytics, Table Managers, Table Metadata Pointers, Schema Evolution, Metadata-Driven Snapshots, Table Schemas, Large-Scale Dataset Management, Manifest-Based File Tracking.

Which projects share features with apache/iceberg?

Projects with overlapping indexed features include: delta-io/delta — Delta is a lakehouse table format that brings ACID transactions and data warehouse consistency to large scale data… apache/hudi — Apache Hudi is an open-source table format that brings ACID transactions, incremental processing, and multi-modal… lancedb/lancedb — LanceDB is a vector database and columnar data store designed to function as a versioned dataset manager and vector… apache/gravitino — Gravitino is a federated metadata lake and unified data catalog designed to manage tables, files, and AI models across… risingwavelabs/risingwave — RisingWave is a cloud-native streaming database and real-time analytics engine that uses standard SQL to process… apache/hive — Apache Hive is a SQL-on-Hadoop data warehouse that enables querying and managing petabytes of data stored in…

Projects sharing features with Iceberg

These projects share indexed features with Iceberg. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.
  • delta-io/deltadelta-io avatar

    delta-io/delta

    8,596View on GitHub↗

    Delta is a lakehouse table format that brings ACID transactions and data warehouse consistency to large scale data lakes on cloud object storage. It serves as an ACID transaction manager, coordinating atomic commits and serializable isolation for concurrent reads and writes across distributed compute engines. The project provides a multi-engine interoperability layer that uses format translation to allow diverse SQL engines and processing frameworks to read and write the same tables. It functions as a data versioning system, utilizing a transaction log to enable time travel, historical snapsh

    Scalaacidanalyticsbig-data
    View on GitHub↗8,596
  • apache/hudiapache avatar

    apache/hudi

    6,097View on GitHub↗

    Apache Hudi is an open-source table format that brings ACID transactions, incremental processing, and multi-modal indexing to data lakes. It provides atomic commits with snapshot isolation, rollback, and optimistic concurrency control for reliable data lake operations, while supporting upserts, record-level updates, and deletions in large analytical datasets. The project distinguishes itself through a timeline-based architecture that coordinates all write operations, enabling features like time-travel querying, incremental change streaming, and multi-modal query views that include snapshot, i

    Javaapacheflinkapachehudiapachespark
    View on GitHub↗6,097
  • lancedb/lancedblancedb avatar

    lancedb/lancedb

    9,031View on GitHub↗

    LanceDB is a vector database and columnar data store designed to function as a versioned dataset manager and vector search engine. It serves as a high-performance backend for indexing and retrieving high-dimensional embeddings, providing the foundation for machine learning data pipelines. The system distinguishes itself through a combination of cloud-native object storage and immutable version tracking, allowing for data time-travel and reproducible AI experiments. It integrates hybrid search capabilities, merging dense vector similarity with BM25 full-text search and SQL-like scalar filters

    HTMLapproximate-nearest-neighbor-searchimage-searchnearest-neighbor-search
    View on GitHub↗9,031
  • apache/gravitinoapache avatar

    apache/gravitino

    2,866View on GitHub↗

    Gravitino is a federated metadata lake and unified data catalog designed to manage tables, files, and AI models across diverse data sources and cloud storage. It serves as a centralized interface for governing schemas, access controls, and tagging across relational databases, messaging queues, and object stores. The project distinguishes itself by unifying the management of AI assets, such as machine learning models and their version lineages, alongside traditional tabular data. It also implements the Iceberg REST specification to provide a standardized metadata server and proxy for lakehouse

    Javaai-catalogdata-catalogdatalake
    View on GitHub↗2,866
Compare all 30 related projects→