Awesome GitHub RepositoriesStream Processing Pipelines

Apache Storm is a distributed stream processing framework and real-time data processing engine. It functions as a fault-tolerant distributed computing system designed to analyze data in motion across a cluster of machines for continuous stream computation. The system enables the creation of fault-tolerant data pipelines and scalable event processing by distributing workloads across a network of computing nodes. This architecture ensures low latency and high throughput for live data while allowing the system to recover automatically from individual node failures. The framework provides capabi

Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

Lazy.js is a JavaScript library that implements a lazy evaluation model for processing collections and data streams. It defers all computation until iteration begins, building chains of transformations that execute only when values are consumed, avoiding intermediate arrays and buffering. The library wraps data sources into a uniform sequence interface, enabling operations like map and filter to be chained together without materializing intermediate results. The library extends lazy processing beyond simple collections to handle asynchronous data sources, DOM events, strings, and Node.js stre

RxGo هي مكتبة برمجة تفاعلية وظيفية وتنفيذ لـ ReactiveX للغة Go. تعمل كمجموعة أدوات لمعالجة التدفق غير المتزامن مصممة لتنسيق البرامج القائمة على الأحداث وتدفقات البيانات باستخدام نمط الملاحظ (observable). تمكن المكتبة من بناء خطوط أنابيب معالجة غير متزامنة تقوم بتحويل وتصفية ودمج تسلسلات الأحداث. وتتميز باستخدام عوامل وظيفية لتكوين خطوط الأنابيب هذه وتوفر آليات لإدارة التنفيذ المتزامن. تغطي مجموعة الأدوات مجموعة واسعة من إمكانات تنسيق التدفق، بما في ذلك تجميع البيانات، ودمج التدفقات المتعددة، وتحويل التدفقات إلى هياكل بيانات ثابتة. وتتضمن دعماً مدمجاً لاستعادة الأخطاء، والتحكم في الضغط العكسي (backpressure) لتنظيم سرعات إنتاج البيانات، وتجميع العمال (worker pooling) لموازاة المعالجة عبر أنوية وحدة المعالجة المركزية.

ZIO is a functional effect system for the JVM that models asynchronous and concurrent programs as pure, composable values with typed error handling and dependency injection. Its core identity is built on fiber-based concurrency, where lightweight, non-blocking fibers execute millions of concurrent tasks with structured lifecycle management, and a dual-channel error model that separates expected business failures from unexpected system defects at compile time. The system provides effect-typed dependency injection through a layer-based dependency graph, pull-based reactive stream processing with

Maxwell is a MySQL change data capture tool and binlog streaming application that converts database modifications into structured JSON events. It functions as a data pipeline that reads MySQL binary logs to synchronize changes across external indices, search engines, and distributed messaging systems such as Kafka. The project provides capabilities to maintain persistent audit trails by recording a chronological history of all database modifications. It enables real-time data synchronization and event-driven architecture integration by streaming database changes to external platforms to trigg