Awesome GitHub RepositoriesData Stream Processors

Pathway is a high-performance data processing framework designed for building unified batch and streaming pipelines. It functions as an orchestrator for complex data transformations, utilizing a differential dataflow engine to process updates incrementally. By treating static datasets and continuous event streams with identical logic, the platform ensures exactly-once processing semantics and consistent results across diverse data sources. The framework distinguishes itself through its specialized support for real-time artificial intelligence and retrieval-augmented generation. It features in

This project serves as a comprehensive technical reference for the architecture and design of data-intensive applications. It provides a structured analysis of the fundamental principles required to build reliable, scalable, and maintainable software systems, covering the core trade-offs inherent in modern data infrastructure. The repository explores the mechanics of distributed data management, including strategies for replication, partitioning, and achieving consensus across multiple nodes. It details the design of storage engines, indexing techniques, and transaction management models, whi

This project is a high-performance MQTT broker and IoT data platform designed to manage millions of concurrent device connections. It provides a scalable infrastructure for ingesting, processing, and routing telemetry data across distributed systems, utilizing an actor-based concurrency model to maintain high availability and state synchronization across cluster nodes. The platform distinguishes itself through integrated stream processing and edge computing capabilities. It allows users to execute declarative SQL-based rules directly against incoming message streams for real-time filtering, t

Benthos is a declarative stream processor and data integration pipeline used to route, transform, and filter information between disparate services. It functions as an at-least-once message broker and change data capture engine, using a transaction model to guarantee message delivery despite system crashes or server faults. The system is defined by an observability-first approach, featuring built-in HTTP health probes, performance metrics export, and distributed request flow tracing. It utilizes a plugin architecture that allows the core engine to be extended with custom binaries for new inpu

Fluent Bit 是一个云原生日志转发器和统一遥测收集器，设计为资源高效的数据流水线。它从多个来源摄取日志、指标和追踪信息，并在将数据路由到外部存储后端之前进行实时处理。 该项目作为一个实时流处理器和 OpenTelemetry 日志处理器，能够使用 SQL 和条件逻辑转换和过滤数据。它还充当分布式追踪代理，可以对追踪进行采样以减少数据量，同时保留完整的请求路径。 该系统通过基于文件系统的缓冲和有状态重试逻辑提供可靠的数据交付，以防止停机期间的数据丢失。其模块化架构支持可插拔的输入和输出插件、元数据驱动的路由，以及通过共享库扩展功能的能力。 该软件可以作为容器部署在不同的 CPU 架构和操作系统上。