Awesome GitHub RepositoriesStream Processing Pipelines

Apache Storm is a distributed stream processing framework and real-time data processing engine. It functions as a fault-tolerant distributed computing system designed to analyze data in motion across a cluster of machines for continuous stream computation. The system enables the creation of fault-tolerant data pipelines and scalable event processing by distributing workloads across a network of computing nodes. This architecture ensures low latency and high throughput for live data while allowing the system to recover automatically from individual node failures. The framework provides capabi

Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

Lazy.js is a JavaScript library that implements a lazy evaluation model for processing collections and data streams. It defers all computation until iteration begins, building chains of transformations that execute only when values are consumed, avoiding intermediate arrays and buffering. The library wraps data sources into a uniform sequence interface, enabling operations like map and filter to be chained together without materializing intermediate results. The library extends lazy processing beyond simple collections to handle asynchronous data sources, DOM events, strings, and Node.js stre

RxGo es una biblioteca de programación reactiva funcional y una implementación de ReactiveX para el lenguaje Go. Sirve como un kit de herramientas de procesamiento de flujos asíncronos diseñado para coordinar programas basados en eventos y flujos de datos utilizando el patrón observable. La biblioteca permite la construcción de pipelines de procesamiento asíncrono que transforman, filtran y combinan secuencias de eventos. Se distingue por el uso de operadores funcionales para componer estos pipelines y proporciona mecanismos para gestionar la ejecución concurrente. El kit de herramientas cubre una amplia gama de capacidades de orquestación de flujos, incluyendo agregación de datos, combinación de múltiples flujos y la conversión de flujos en estructuras de datos estáticas. Incluye soporte integrado para recuperación de errores, control de contrapresión (backpressure) para regular las velocidades de producción de datos y agrupación de workers para paralelizar el procesamiento a través de núcleos de CPU.

ZIO is a functional effect system for the JVM that models asynchronous and concurrent programs as pure, composable values with typed error handling and dependency injection. Its core identity is built on fiber-based concurrency, where lightweight, non-blocking fibers execute millions of concurrent tasks with structured lifecycle management, and a dual-channel error model that separates expected business failures from unexpected system defects at compile time. The system provides effect-typed dependency injection through a layer-based dependency graph, pull-based reactive stream processing with

Maxwell is a MySQL change data capture tool and binlog streaming application that converts database modifications into structured JSON events. It functions as a data pipeline that reads MySQL binary logs to synchronize changes across external indices, search engines, and distributed messaging systems such as Kafka. The project provides capabilities to maintain persistent audit trails by recording a chronological history of all database modifications. It enables real-time data synchronization and event-driven architecture integration by streaming database changes to external platforms to trigg