treeverse/lakeFS

Features

• Data Versioning - Implements Git-like versioning for data lakes, using branches, reproducible snapshots, and atomic merges.
• Zero-Copy Data Branching - Enables the creation of lightweight, isolated environments for data experimentation without duplicating physical data files.
• S3-Compatible Storage Adapters - Provides an S3-compatible gateway interface for read and write operations via standard cloud storage protocols.
• Data Versioning Systems - Provides a version control layer for object storage with Git-like branching and commits for large datasets.
• Table Versioning - Provides a complete history of modifications for Iceberg tables by staging changes on specific references.
• Ingestion Isolation - Isolates incoming data on separate branches to validate quality before merging into the main dataset.
• Data Quality Frameworks - Provides a framework for validating datasets against governance policies and quality tests before production merging.
• Dataset Versioning Platforms - Tracks and manages historical versions of large-scale research datasets and models to ensure reproducibility.
• Atomic Transactional Commits - Groups multiple data operations into discrete atomic units that can be committed or rolled back to ensure consistency.
• Iceberg REST Catalog Servers - Implements a standardized Iceberg REST catalog server for metadata management and table access.
• Immutable Commit Versioning - Captures the current state of a branch as a unique, immutable commit for auditing and reproducibility.
• Metadata-Layer Versioning - Tracks object state via a separate metadata layer in a database instead of modifying the storage backend.
• Object Versioning - Applies version control semantics to data lakes to enable repeatable and atomic operations.
• Versioned Datasets - Enables the creation and saving of named collections of data tracked through a version control system.
• Versioned Repository Provisioning - Establishes a versioned storage namespace linked to an object storage location to track history.
• Zero-Copy Cloning - Creates lightweight isolated namespaces for datasets without duplicating the underlying physical data files.
• Commit Finalization with Messages - Records snapshots of current changes to a branch with descriptive messages for traceability.
• Data Branching & Merging - Manages the table lifecycle by isolating changes in separate branches before atomic merging.
• Git Branching & Merging - Manages parallel data development streams using branches and provides mechanisms to merge changes between them.
• Dataset Branching - Creates isolated namespaces for datasets to enable parallel experimentation without affecting production data.
• Cloud Storage - Manages data residing across different on-premises, hybrid, or multi-cloud storage locations.
• S3 Gateway Interfaces - Provides an S3-compatible gateway that routes standard cloud storage protocols to versioned data repositories.
• Repository Management APIs - Provides HTTP endpoints for administrative operations such as branch creation and data commits.
• Zero-Copy Branching - Generates zero-copy branches of production data to test pipelines without affecting downstream consumers.
• Atomic Commit Pipelines - Groups multiple object changes into a single immutable snapshot to ensure data consistency and reproducibility.
• Event-Driven Hook Systems - Executes custom scripts or webhooks in response to repository events to automate validation and notifications.
• Merge Requirement Governance - Ensures data meets strict governance and schema requirements via quality gates before merging into production.
• Quality Gates - Implements quality gates that validate data against governance policies before publishing changes to production.
• Data Lineage - Tracks the transformation history and provenance of datasets to ensure transparency in experiments.
• Experiment Metadata Tracking - Records model performance and data provenance by attaching custom metrics to specific commits.
• ML Platform Integrations - Connects machine learning workflows to a versioned data lake by overriding S3 endpoints for Amazon SageMaker.
• Embedding Versioning - Captures embeddings, raw data, and metadata as a single atomic commit for consistency.
• Data-Code Version Linking - Links dataset versions to corresponding code commits to ensure reproducible pipelines and models.
• Azure Blob Storage Integration - Interfaces with Azure Blob and Data Lake storage to manage data as a versioned repository.
• Big Data Compute Integrations - Integrates data versioning with Spark environments using a custom Hadoop FileSystem implementation.
• Table Querying - Performs SQL queries against Iceberg tables using a REST catalog to maintain versioning.
• Catalog Integration - Synchronizes table metadata to external query engines and warehouses for SQL-based access.
• Federated Catalogs - Exposes managed tables to external cloud services via a real-time federated metadata catalog.
• Table Versioning - Maintains versioned views of underlying data and associated metadata specifically for Delta Lake tables.
• Versioned Data Restorations - Reverts datasets to a specific historical point in time to resolve critical data errors.
• Differential Dataset Comparisons - Calculates differences between dataset versions by comparing commit diffs and versioned URIs.
• Data Science Library Integrations - Enables loading and saving dataframes directly to and from libraries like pandas and polars using a specialized URI scheme.
• Historical - Tracks the state of data over time to reproduce historical environments for debugging and ML validation.
• Cross-Engine Data Synchronization - Moves large datasets between different object store providers using distributed synchronization utilities.
• Pipeline-Integrated Versioning - Integrates version control operations like branching and commits directly as discrete steps within machine learning pipelines.
• Data Warehouse Integrations - Integrates with data warehouse software to manage large datasets in object storage using SQL.
• Delta Lake Table Management - Provides the ability to read and write Delta Lake tables using compatible URIs and credentials.
• Federated Data Query Engines - Executes SQL queries against data lake tables via catalog federation without copying data.
• Partial Read Prevention - Blocks access to data from failed ingestion processes to ensure consumers only see complete datasets.
• Branch-Based Isolation - Prevents data corruption by isolating concurrent modifications in dedicated branches before performing atomic merges.
• Repository Cloning - Creates a duplicate of an existing repository by copying metadata and data to a new environment.
• Storage Garbage Collection - Reclaims storage space by identifying and deleting stale objects and branches based on defined rules.
• Streaming Source and Sink Integration - Exports high-throughput data feeds from streaming platforms into versioned object stores via S3 sinks.
• Transactional Replication - Mirrors repositories across locations by advancing state only after all data for a commit is fully copied.
• Change Proposal Workflows - Provides structured mechanisms for proposing and reviewing data changes before they are integrated into the main branch.
• Event-Driven Automation Hooks - Runs embedded scripts automatically when specific actions occur to validate data or trigger notifications.
• Transactional Mirrors - Replicates repositories into consistent, read-only copies across remote locations for high availability.
• Dataset Diffing - Analyzes differences between branches or tags by querying multiple version-specific catalogs in a single session.
• Pre-Commit Quality Hooks - Triggers validation tests during commit operations to prevent low-quality data from reaching production.
• Repository State Restoration - Reconstructs a repository by reloading commits, tags, and branches from a metadata manifest.
• Catalog Synchronizations - Pushes table metadata to or pulls updates from external catalogs to keep environments in sync.
• Version Tag Management - Creates immutable pointers to specific commits to mark stable releases or milestones.
• Dataset Versioning - Logs specific dataset commits as inputs to machine learning experiments for reproducibility.
• Cross-Cloud Repository Management - Centralizes the administration of repositories spanning multiple cloud providers and on-premises environments.
• Branch Protections - Ensures data integrity by restricting direct uploads and deletions on protected branches.
• Role-Based Access Control - Manages permissions for table and namespace operations using role-based access controls.
• Reproducible Data Transformations - Tracks data transformations from raw sets to final versions to ensure repeatable experiments.
• Team Collaboration Tools - Facilitates team collaboration by allowing isolated data changes to be shared and reviewed via a pull request workflow.
• User Action Audit Logging - Maintains detailed audit logs of user actions performed on specific resources.
• Audit Logging Systems - Logs a detailed record of user actions and resource changes for security and compliance tracking.
• Object Storage Backups - Exports metadata and copies object storage to separate locations to ensure disaster recovery.
• Versioned Result Reproduction - Recreates exact inference results by linking queries to specific commit IDs, tags, or branches.
• Data Lake Management - Platform providing Git-like version control for object-storage lakes.
• Datenmanagement - Versioned data lake on top of object storage.
• Data Orchestration - Version control for data lakes on object storage.
• Experiment and Data Management - Versioned data lake management on top of object storage.
• File Storage Systems - Git-like version control for object storage.
• Data Engineering Tools - Tool for versioning and managing data lake operations and ETL jobs.

Star-Verlauf