Awesome GitHub RepositoriesData Stream Processors

Pathway is a high-performance data processing framework designed for building unified batch and streaming pipelines. It functions as an orchestrator for complex data transformations, utilizing a differential dataflow engine to process updates incrementally. By treating static datasets and continuous event streams with identical logic, the platform ensures exactly-once processing semantics and consistent results across diverse data sources. The framework distinguishes itself through its specialized support for real-time artificial intelligence and retrieval-augmented generation. It features in

This project serves as a comprehensive technical reference for the architecture and design of data-intensive applications. It provides a structured analysis of the fundamental principles required to build reliable, scalable, and maintainable software systems, covering the core trade-offs inherent in modern data infrastructure. The repository explores the mechanics of distributed data management, including strategies for replication, partitioning, and achieving consensus across multiple nodes. It details the design of storage engines, indexing techniques, and transaction management models, whi

This project is a high-performance MQTT broker and IoT data platform designed to manage millions of concurrent device connections. It provides a scalable infrastructure for ingesting, processing, and routing telemetry data across distributed systems, utilizing an actor-based concurrency model to maintain high availability and state synchronization across cluster nodes. The platform distinguishes itself through integrated stream processing and edge computing capabilities. It allows users to execute declarative SQL-based rules directly against incoming message streams for real-time filtering, t

Benthos is a declarative stream processor and data integration pipeline used to route, transform, and filter information between disparate services. It functions as an at-least-once message broker and change data capture engine, using a transaction model to guarantee message delivery despite system crashes or server faults. The system is defined by an observability-first approach, featuring built-in HTTP health probes, performance metrics export, and distributed request flow tracing. It utilizes a plugin architecture that allows the core engine to be extended with custom binaries for new inpu

Fluent Bit ist ein Cloud-nativer Log-Shipper und einheitlicher Telemetrie-Collector, der als ressourceneffiziente Datenpipeline konzipiert ist. Er nimmt Logs, Metriken und Traces aus mehreren Quellen auf und verarbeitet sie in Echtzeit, bevor die Daten an externe Speicher-Backends weitergeleitet werden. Das Projekt fungiert als Echtzeit-Stream-Prozessor und OpenTelemetry-Log-Prozessor, der in der Lage ist, Daten mithilfe von SQL und bedingter Logik zu transformieren und zu filtern. Es fungiert zudem als verteilter Tracing-Agent, der Traces sampeln kann, um das Datenvolumen zu reduzieren und gleichzeitig vollständige Anforderungspfade zu bewahren. Das System bietet eine zuverlässige Datenübermittlung durch dateisystembasiertes Buffering und zustandsbehaftete Wiederholungslogik, um Datenverlust bei Ausfällen zu verhindern. Seine modulare Architektur unterstützt steckbare Eingabe- und Ausgabe-Plugins, metadatengesteuertes Routing und die Möglichkeit, Funktionen über gemeinsam genutzte Bibliotheken zu erweitern. Die Software kann als Container über verschiedene CPU-Architekturen und Betriebssysteme hinweg bereitgestellt werden.