1 repository
Awesome GitHub RepositoriesBig Data Compute Integrations
Filesystem implementations and connectors that bridge versioned storage with distributed compute engines.
Distinct from Apache Spark Pipelines: Candidates focused on deep learning or pipeline environments, not the Hadoop FileSystem bridge for data versioning.
Explore 1 awesome GitHub repository matching data & databases · Big Data Compute Integrations. Refine with filters or upvote what's useful.
Awesome Big Data Compute Integrations GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- treeverse/lakefs
treeverse/lakeFS
5,406Vezi pe GitHublakeFS is a data lake versioning system that provides Git-like branching and commits for large datasets stored in object storage. It functions as a version control layer, enabling the creation of immutable snapshots, atomic commits, and zero-copy branching to create isolated environments for data experimentation without duplicating physical files. The system serves as an S3-compatible storage gateway and an Iceberg REST catalog, allowing standard cloud storage protocols and compatible clients to manage versioned tables. It acts as a data quality gatekeeper by using an event-driven hook system
Integrates data versioning with Spark environments using a custom Hadoop FileSystem implementation.
Go
