Awesome GitHub RepositoriesData Orchestration

Pathway is a high-performance data processing framework designed for building unified batch and streaming pipelines. It functions as an orchestrator for complex data transformations, utilizing a differential dataflow engine to process updates incrementally. By treating static datasets and continuous event streams with identical logic, the platform ensures exactly-once processing semantics and consistent results across diverse data sources. The framework distinguishes itself through its specialized support for real-time artificial intelligence and retrieval-augmented generation. It features in

Cadence is a distributed workflow orchestration engine designed to execute long-running, asynchronous business logic with built-in durability and resilience across distributed systems. It functions as a stateful process manager that ensures processes resume from their last known state following system crashes or network outages. The platform utilizes a distributed task queue to manage work across independent worker nodes and supports persistence via SQL or Cassandra backend storage. It includes a workflow visualization dashboard for inspecting execution histories and state traces, alongside a

lakeFS 是一个数据湖版本控制系统，为存储在对象存储中的大型数据集提供类似 Git 的分支和提交功能。它作为一个版本控制层，支持创建不可变快照、原子提交和零拷贝分支，从而在不复制物理文件的情况下为数据实验创建隔离环境。 该系统充当 S3 兼容的存储网关和 Iceberg REST 目录，允许标准云存储协议和兼容客户端管理版本化表。它通过使用事件驱动的钩子系统在更改合并到生产环境之前根据治理策略验证数据集，从而充当数据质量守门人。 该平台涵盖了广泛的数据治理功能，包括 Pull Request 协作、基于角色的访问控制和数据血缘追踪。它为工作流编排、机器学习管线和各种大数据计算引擎提供了集成，支持多云存储连接以及通过 SSO 和 SCIM 进行身份同步。 该软件可以使用二进制文件、容器或 Helm Chart 安装，以便在 Kubernetes 上部署。

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

Substation is a toolkit for routing, normalizing, and enriching security event and audit logs.

Model Context Protocol (MCP) Server for the Keboola Platform