Awesome GitHub RepositoriesSQL-Based Pipeline Definitions

RisingWave is a cloud-native streaming database and real-time analytics engine that uses standard SQL to process continuous data streams. It functions as a streaming data lakehouse, combining the capabilities of a streaming SQL database with a platform that integrates streaming ingestion with open table formats. The system is distinguished by its use of the PostgreSQL wire protocol, allowing it to integrate with existing SQL tools and drivers. It employs a decoupled compute and storage architecture, persisting streaming state and materialized views in cloud object storage to enable independen

Arroyo is a high-performance stream processing platform built in Rust. It executes continuous SQL queries on streaming data with event-time semantics, enabling accurate windowed aggregations, joins, and stateful computations on unbounded event streams. The platform uses native Rust execution for high throughput and low latency, with periodic checkpointing for exactly-once fault tolerance and horizontal scaling across distributed workers. The system integrates deeply with Kafka for reading and writing topics with exactly-once delivery and supports change data capture (CDC) from MySQL and Postg

Chunjun 是一个分布式数据集成框架和基于 SQL 的 ETL 流水线，旨在实现异构数据源之间的数据同步。它作为一款变更数据捕获（CDC）工具和异构数据同步器，利用分布式处理环境在不同数据库类型之间迁移和转换数据。 该系统的特色在于其基于插件的连接器架构，允许开发自定义源和目标插件，以扩展对非原生支持数据系统的连接。它支持从关系型数据库日志中进行实时变更数据捕获，并实现模式演进传播，自动将结构变更从源表应用到目标表。 该框架提供了增量数据同步和使用 SQL 逻辑进行跨源数据计算的能力。可靠性通过基于检查点的任务恢复机制来管理，以恢复中断的传输，并利用死信队列进行脏数据管理，以审计格式错误的数据记录。 集成任务可部署在独立集群、Yarn 或 Kubernetes 环境中，并支持通过 Docker 进行容器化部署。