Awesome GitHub RepositoriesStreaming SQL Transformations

SeaTunnel is a distributed data integration engine designed to synchronize structured and unstructured data across diverse sources and sinks. It functions as a multi-engine execution framework that can run data integration tasks across different distributed computing backends to optimize workload performance. The project is distinguished by a visual data pipeline designer for configuring workflows without manual code and a specialized change data capture tool for streaming incremental database updates. It also includes an enrichment pipeline that integrates large language models and embedding

RisingWave is a cloud-native streaming database and real-time analytics engine that uses standard SQL to process continuous data streams. It functions as a streaming data lakehouse, combining the capabilities of a streaming SQL database with a platform that integrates streaming ingestion with open table formats. The system is distinguished by its use of the PostgreSQL wire protocol, allowing it to integrate with existing SQL tools and drivers. It employs a decoupled compute and storage architecture, persisting streaming state and materialized views in cloud object storage to enable independen

Fluent Bit 是一个云原生日志转发器和统一遥测收集器，设计为资源高效的数据流水线。它从多个来源摄取日志、指标和追踪信息，并在将数据路由到外部存储后端之前进行实时处理。 该项目作为一个实时流处理器和 OpenTelemetry 日志处理器，能够使用 SQL 和条件逻辑转换和过滤数据。它还充当分布式追踪代理，可以对追踪进行采样以减少数据量，同时保留完整的请求路径。 该系统通过基于文件系统的缓冲和有状态重试逻辑提供可靠的数据交付，以防止停机期间的数据丢失。其模块化架构支持可插拔的输入和输出插件、元数据驱动的路由，以及通过共享库扩展功能的能力。 该软件可以作为容器部署在不同的 CPU 架构和操作系统上。

Materialize is a streaming SQL database that continuously ingests live data from sources such as Kafka, Redpanda, PostgreSQL, and MySQL, and incrementally maintains materialized views. It provides a PostgreSQL-compatible query engine that accepts standard SQL over the PostgreSQL wire protocol, enabling any existing SQL client or BI tool to query real-time data. The system also includes a Model Context Protocol (MCP) server that exposes live materialized view data to AI agents, providing fresh context without polling. Materialize distinguishes itself through its ability to offer configurable c

Arroyo is a high-performance stream processing platform built in Rust. It executes continuous SQL queries on streaming data with event-time semantics, enabling accurate windowed aggregations, joins, and stateful computations on unbounded event streams. The platform uses native Rust execution for high throughput and low latency, with periodic checkpointing for exactly-once fault tolerance and horizontal scaling across distributed workers. The system integrates deeply with Kafka for reading and writing topics with exactly-once delivery and supports change data capture (CDC) from MySQL and Postg

StreamPark 是一个集中式管理平台，旨在协调分布式流处理和批处理应用程序的部署、监控和操作生命周期。它作为一个数据流水线控制平面和编排器，专门为 Apache Flink 和 Hadoop YARN 环境提供管理功能。 该平台通过任务部署的低代码方法和支持多种处理运行时的多引擎执行适配器脱颖而出。它通过将流式 SQL 分析与处理版本控制、二进制上传和基于保存点 (savepoint) 的状态恢复的资源驱动部署流水线相结合，促进了实时数据流水线管理。 该系统涵盖了广泛的功能，包括分布式作业编排、通过预构建连接器进行的实时数据集成，以及通过 LDAP 或 SSO 进行的身份集成。它还提供用于秒级应用程序监控和自动化操作故障通知的可观测性工具。

Chunjun 是一个分布式数据集成框架和基于 SQL 的 ETL 流水线，旨在实现异构数据源之间的数据同步。它作为一款变更数据捕获（CDC）工具和异构数据同步器，利用分布式处理环境在不同数据库类型之间迁移和转换数据。 该系统的特色在于其基于插件的连接器架构，允许开发自定义源和目标插件，以扩展对非原生支持数据系统的连接。它支持从关系型数据库日志中进行实时变更数据捕获，并实现模式演进传播，自动将结构变更从源表应用到目标表。 该框架提供了增量数据同步和使用 SQL 逻辑进行跨源数据计算的能力。可靠性通过基于检查点的任务恢复机制来管理，以恢复中断的传输，并利用死信队列进行脏数据管理，以审计格式错误的数据记录。 集成任务可部署在独立集群、Yarn 或 Kubernetes 环境中，并支持通过 Docker 进行容器化部署。