Awesome GitHub RepositoriesStreaming Processors

Apache Flink is a distributed processing engine designed for both high-throughput, low-latency data streams and finite batch workloads. It functions as a stateful stream processor and a SQL stream processing engine, providing a unified runtime to execute relational queries and event-based transformations. The system is distinguished by its ability to manage persistent operator state to ensure exactly-once processing guarantees and consistency during failures. It features specialized capabilities for complex event processing to detect temporal patterns and handles out-of-order events using eve

Toon is a data serialization library and toolkit designed to convert complex objects into compact, human-readable formats optimized for large language models. By focusing on token efficiency, the library minimizes the context window footprint of structured data through techniques like key folding and tabular layout optimization. It provides a streaming-capable processor that handles the encoding and decoding of hierarchical data while maintaining structural integrity. The project distinguishes itself through its path-aware transformation pipeline and configurable serialization logic, which al

This project is a feature-rich Go client library designed for interacting with Redis. It serves as a comprehensive interface for managing remote data stores, enabling developers to execute standard database commands, handle complex data structures, and perform asynchronous operations within Go applications. The library distinguishes itself through its support for advanced Redis capabilities, including connection pooling, pipelining, and transactional integrity. It provides specialized primitives for managing distributed clusters, including automated topology updates and request routing to sha

ThingsBoard is an IoT device management platform designed for provisioning, monitoring, and managing large fleets of hardware devices and assets across multiple customers. It functions as a microservices infrastructure that allows the deployment of data collection and management services as independent containerized units for scaling. The platform includes a rule-based stream processor that transforms incoming device data and triggers alarms using customizable rule chains. It also provides a data visualization suite consisting of dashboards and widgets to display real-time telemetry and syste

Age is a command-line utility for file encryption that utilizes hybrid cryptography to secure data for multiple recipients. It employs a combination of asymmetric key exchange and symmetric encryption to protect files, supporting access control through public keys, shared passphrases, and hardware-backed identity integration. The tool is designed for memory-efficient operation, utilizing stream-oriented processing to handle large datasets in small, sequential chunks. It features a stanza-based metadata framing system that allows for extensible file headers and supports random-access decryptio

ioredis is a performance-focused Redis client for Node.js designed to execute commands and manage data connections. It provides a specialized interface for interacting with standalone servers, sharded clusters, and high-availability setups. The library distinguishes itself with native support for Redis Cluster, featuring automatic slot discovery and network address mapping, and Redis Sentinel for master node discovery and automatic failover. It also includes a dedicated Lua scripting interface that utilizes server-side caching to ensure atomic operations. The project covers a broad set of ca

ioredis is a performance-focused Redis client for Node.js applications. It provides a comprehensive interface for interacting with Redis servers, including specialized clients for sharded clusters and Sentinel-based high availability environments. The project distinguishes itself through advanced networking and execution capabilities, such as automatic event-loop pipelining to reduce overhead and a system for routing read-write traffic between primary and replica nodes. It also features a dedicated Lua scripting interface that allows server-side scripts to be registered as custom client comma

CryptoSwift is a cryptography library implemented entirely in the Swift programming language. It provides a collection of standard cryptographic algorithms for encryption, decryption, and hashing without relying on native C libraries or system frameworks. The library supports symmetric and asymmetric encryption, including RSA key generation and signature management. It features authenticated encryption schemes and the ability to generate cryptographic digests for data integrity verification. The toolset covers message authentication codes, secure key derivation from passwords, and data paddi

phpredis is a C-based native extension that bridges PHP applications with Redis servers for high-performance data storage and retrieval. It serves as an interface for manipulating strings, hashes, lists, sets, and sorted sets while providing a direct path for executing Redis commands and server-side scripts. The extension provides comprehensive support for distributed environments and high availability. It interfaces with Redis Cluster to distribute data across multiple nodes using hash slots and manages Redis Sentinel for service discovery and automatic failover. It also enables shared state

Redis is a high-performance in-memory key-value store that functions as a distributed cache, message broker, and NoSQL database. It provides sub-millisecond read and write access to data stored in RAM and can operate as a vector database for indexing high-dimensional embeddings. The system supports a wide range of data storage and synchronization primitives, including the management of strings, hashes, lists, sets, and JSON documents. It enables real-time data operations through atomic transactions, hybrid persistence using snapshots and append-only logs, and high-availability configurations

Watermill is a Go library for building event-driven applications. It provides a framework for implementing event-driven architectures through a consistent interface for message publishing and subscribing, featuring a pub/sub message router and event stream processing pipelines. The library uses interface-based abstractions to decouple message producers and consumers from the underlying message broker. This allows different messaging technologies to be swapped via plugin-based broker integration. It employs a router-driven dispatch system and middleware-based pipelines to manage cross-cutting

RedisInsight is a graphical user interface and management tool for browsing, analyzing, and administering Redis databases. It provides a visual environment for exploring key-value data structures, managing database instances, and performing data analysis across different operating systems and deployments. The tool distinguishes itself by providing dedicated visual managers for complex operations, including a vector database manager for configuring embeddings and similarity searches, a query workbench for executing raw commands and Lua scripts, and a performance monitoring dashboard for tracki

Enterprise job scheduling middleware with distributed computing ability.

Faust is a Python library for building distributed stream processing applications that integrate with Kafka. It functions as an asynchronous stream processor designed to handle high-throughput event streams and real-time data analysis using asynchronous functions. The system operates as a distributed stream processor and state store, utilizing sharding and partitioned topics to scale processing workloads horizontally across multiple worker nodes. It maintains state through a replicated key-value storage system backed by local databases to ensure high availability and fast recovery. The frame

Feast is an open-source feature store for machine learning that provides a central platform for defining, storing, and serving features across both training and inference workflows. It operates as a declarative system where feature definitions are written as code in Python files, synchronized to a central registry, and made available for low-latency online retrieval or point-in-time correct historical joins for training datasets. The project abstracts storage behind a pluggable architecture, allowing offline and online backends to be swapped without changing retrieval logic, and coordinates ma

Octosql 是一个联邦 SQL 查询引擎、数据转换器和流式 SQL 处理器。它允许用户跨多个异构数据源（包括不同类型的数据库和文件格式）执行单一 SQL 语句，从而合并并转换结果集。 该系统的独特之处在于将 CSV、JSONLines 和 Parquet 文件视为虚拟表，并利用基于插件的架构扩展对外部存储引擎的连接。它作为无限数据流的流式处理器，使用水印（watermarks）、撤回（retractions）和翻滚窗口（tumbling windows）来维持乱序事件的一致性。此外，它还可用作 SQL 数据生成器，通过表值函数生成合成数据集和记录流。 该引擎具备跨源数据连接和多源分析能力，并通过源端谓词下推（predicate push-down）进行优化，以减少数据传输。它通过包含联合类型的静态类型系统管理复杂数据，并提供查询执行计划可视化功能以增强可观测性。

Riemann 是一个基于 Clojure 的事件流处理器和实时分析引擎。它作为一个网络遥测管道和可扩展事件路由器，用于摄取、转换和路由来自分布式系统的事件数据。 该系统使用领域特定语言来计算连续流上的指标和统计模式，从而实现网络趋势分析和实时警报。它支持从类路径动态加载插件，并允许在不中断活动事件流的情况下实时重新加载配置。 功能包括集中式遥测聚合、事件元数据标记和有状态事件索引。该系统通过拆分、批处理和过滤处理事件流的调度，同时通过加密和身份验证提供安全的网络传输。

RealtimeTTS 是一个实时文本转语音引擎和流处理器，旨在以最小延迟将文本或令牌流转换为音频播放。它提供了一个用于管理音频流、合成进度以及集成本地或云端语音引擎的编程接口。 该系统包括一个神经语音克隆工具，通过从参考音频样本中提取声学特征来生成合成语音。它利用基于提供程序的抽象来跨不同的神经模型和云 API 路由合成请求。 该项目涵盖了一系列功能领域，包括异步播放控制、令牌流音频合成以及在单词或字符级别触发回调的语音同步工作流。它还支持将合成音频直接导出为 WAV 文件以进行持久存储。