Awesome GitHub RepositoriesData Versioning

DataX is a distributed data integration framework and plugin-based ETL tool designed for synchronizing large datasets between heterogeneous sources and destinations. It functions as a JDBC data migration engine and offline synchronization tool, enabling the movement of data between relational databases, NoSQL stores, and object storage. The system utilizes a plugin-based connector architecture that decouples reader and writer logic, allowing it to map and transform data types across different storage engines using a standardized internal representation. This design supports heterogeneous data

Presto is a distributed SQL query engine designed for high-performance analytical processing across heterogeneous data sources. It functions as a data federation platform and massively parallel processing engine, allowing users to execute interactive queries against diverse storage systems without requiring data migration. By mapping remote metadata and structures to a unified relational namespace, it enables seamless cross-platform analysis through a standard SQL interface. The engine distinguishes itself through a pluggable connector architecture and a shared-nothing distributed processing

Databend is a cloud-native data warehouse and OLAP database designed for large-scale analytics. It functions as a SQL-compliant engine and serverless analytics platform that separates compute from storage to allow for independent scaling. The system integrates vector database capabilities, indexing high-dimensional embeddings to enable semantic, hybrid, and full-text searches across massive datasets. It further distinguishes itself through serverless compute management that automatically scales resources based on demand and shuts them down during idle periods. The platform covers a broad set

LanceDB is a vector database and columnar data store designed to function as a versioned dataset manager and vector search engine. It serves as a high-performance backend for indexing and retrieving high-dimensional embeddings, providing the foundation for machine learning data pipelines. The system distinguishes itself through a combination of cloud-native object storage and immutable version tracking, allowing for data time-travel and reproducible AI experiments. It integrates hybrid search capabilities, merging dense vector similarity with BM25 full-text search and SQL-like scalar filters

Delta is a lakehouse table format that brings ACID transactions and data warehouse consistency to large scale data lakes on cloud object storage. It serves as an ACID transaction manager, coordinating atomic commits and serializable isolation for concurrent reads and writes across distributed compute engines. The project provides a multi-engine interoperability layer that uses format translation to allow diverse SQL engines and processing frameworks to read and write the same tables. It functions as a data versioning system, utilizing a transaction log to enable time travel, historical snapsh

Enso is a visual dataflow programming environment and multi-language data processing engine that compiles Enso, Python, Java, and JavaScript into a unified representation with a shared memory model for zero-overhead inter-language calls. It functions as a self-service data preparation and analysis platform where users can build data pipelines by connecting nodes in a graph, switching between a no-code visual interface and a code view while keeping all changes reviewable. The platform also serves as a cloud data workflow scheduler and API exposer, allowing workflows to run on a timetable or be

Noms is a distributed version control database and content-addressable data store. It identifies data by cryptographic hashes to ensure integrity and deduplication, while tracking dataset state changes through a sequence of immutable commits to enable branching, forking, and historical recovery. The system functions as a peer-to-peer data synchronizer, reconciling state between disconnected database instances to ensure all nodes converge on the same data. It distinguishes itself as a schema-flexible document store that supports self-describing types, allowing schemas to evolve and widen as ne

bup is a deduplicating backup manager and incremental backup system. It uses a Git packfile-based storage format to eliminate redundant data across files and versions, treating every incremental save as a full backup. The system provides secure remote transport interfaces for transferring and managing backup data on remote servers via SSH. It also includes a backup repository browser available as both a web interface and a filesystem mount for exploring and retrieving files from snapshots. The project covers broad capability areas including disaster recovery, repository administration, and s

pgroll is a PostgreSQL migration framework designed for zero-downtime schema changes. It applies non-blocking DDL operations that avoid exclusive locks on tables, and uses trigger-based column backfill to populate new columns while keeping them synchronized with old ones. The framework wraps each migration step in a database transaction that can be atomically committed or rolled back, and creates a versioned view layer that exposes both old and new schema versions simultaneously to client applications. The tool distinguishes itself by managing multiple schema versions via views, enabling safe

lakeFS 是一个数据湖版本控制系统，为存储在对象存储中的大型数据集提供类似 Git 的分支和提交功能。它作为一个版本控制层，支持创建不可变快照、原子提交和零拷贝分支，从而在不复制物理文件的情况下为数据实验创建隔离环境。 该系统充当 S3 兼容的存储网关和 Iceberg REST 目录，允许标准云存储协议和兼容客户端管理版本化表。它通过使用事件驱动的钩子系统在更改合并到生产环境之前根据治理策略验证数据集，从而充当数据质量守门人。 该平台涵盖了广泛的数据治理功能，包括 Pull Request 协作、基于角色的访问控制和数据血缘追踪。它为工作流编排、机器学习管线和各种大数据计算引擎提供了集成，支持多云存储连接以及通过 SSO 和 SCIM 进行身份同步。 该软件可以使用二进制文件、容器或 Helm Chart 安装，以便在 Kubernetes 上部署。

RStudio is a specialized integrated development environment for the R programming language and statistical computing. It provides a workbench for writing, debugging, and executing R code, offering both a desktop application and a server-hosted collaborative platform for managing data science projects. The platform enables the creation of interactive data applications, AI-powered dashboards, and technical reports. It facilitates the sharing of analysis results through a centralized publishing platform and supports the rendering of notebooks and markdown into multiple file formats. The environ