Awesome GitHub RepositoriesStream Processing Pipelines

Apache Storm is a distributed stream processing framework and real-time data processing engine. It functions as a fault-tolerant distributed computing system designed to analyze data in motion across a cluster of machines for continuous stream computation. The system enables the creation of fault-tolerant data pipelines and scalable event processing by distributing workloads across a network of computing nodes. This architecture ensures low latency and high throughput for live data while allowing the system to recover automatically from individual node failures. The framework provides capabi

Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

Lazy.js is a JavaScript library that implements a lazy evaluation model for processing collections and data streams. It defers all computation until iteration begins, building chains of transformations that execute only when values are consumed, avoiding intermediate arrays and buffering. The library wraps data sources into a uniform sequence interface, enabling operations like map and filter to be chained together without materializing intermediate results. The library extends lazy processing beyond simple collections to handle asynchronous data sources, DOM events, strings, and Node.js stre

RxGo 是一个函数式响应式编程库，也是 ReactiveX 的 Go 语言实现。它作为一个异步流处理工具包，旨在利用可观察模式协调基于事件的程序和数据流。 该库支持构建异步处理管道，以转换、过滤和组合事件序列。其特色在于使用函数式操作符来组合这些管道，并提供了管理并发执行的机制。 该工具包涵盖了广泛的流编排功能，包括数据聚合、多流组合以及将流转换为静态数据结构。它内置了对错误恢复、用于调节数据生产速度的背压控制以及用于跨 CPU 核心并行处理的工作池的支持。

ZIO is a functional effect system for the JVM that models asynchronous and concurrent programs as pure, composable values with typed error handling and dependency injection. Its core identity is built on fiber-based concurrency, where lightweight, non-blocking fibers execute millions of concurrent tasks with structured lifecycle management, and a dual-channel error model that separates expected business failures from unexpected system defects at compile time. The system provides effect-typed dependency injection through a layer-based dependency graph, pull-based reactive stream processing with

Maxwell 是一个 MySQL 变更数据捕获 (CDC) 工具和 binlog 流处理应用程序，可将数据库修改转换为结构化 JSON 事件。它作为一个数据管道，读取 MySQL 二进制日志，以便在外部索引、搜索引擎和 Kafka 等分布式消息系统中同步变更。 该项目提供了通过记录所有数据库修改的时间顺序历史来维护持久审计跟踪的功能。它通过将数据库变更流式传输到外部平台来触发工作流并通知微服务，从而实现实时数据同步和事件驱动架构集成。 该系统涵盖了广泛的功能领域，包括通过初始快照进行数据引导、模式版本管理和事件过滤。它通过分区键路由进行流量管理，并提供通过 HTTP 端点暴露的健康检查和性能指标监控。 与数据库和流处理生产者的连接使用 SSL 和加密通信进行保护。