Awesome GitHub RepositoriesStreaming Processors

Apache Flink is a distributed processing engine designed for both high-throughput, low-latency data streams and finite batch workloads. It functions as a stateful stream processor and a SQL stream processing engine, providing a unified runtime to execute relational queries and event-based transformations. The system is distinguished by its ability to manage persistent operator state to ensure exactly-once processing guarantees and consistency during failures. It features specialized capabilities for complex event processing to detect temporal patterns and handles out-of-order events using eve

Toon is a data serialization library and toolkit designed to convert complex objects into compact, human-readable formats optimized for large language models. By focusing on token efficiency, the library minimizes the context window footprint of structured data through techniques like key folding and tabular layout optimization. It provides a streaming-capable processor that handles the encoding and decoding of hierarchical data while maintaining structural integrity. The project distinguishes itself through its path-aware transformation pipeline and configurable serialization logic, which al

This project is a feature-rich Go client library designed for interacting with Redis. It serves as a comprehensive interface for managing remote data stores, enabling developers to execute standard database commands, handle complex data structures, and perform asynchronous operations within Go applications. The library distinguishes itself through its support for advanced Redis capabilities, including connection pooling, pipelining, and transactional integrity. It provides specialized primitives for managing distributed clusters, including automated topology updates and request routing to sha

ThingsBoard is an IoT device management platform designed for provisioning, monitoring, and managing large fleets of hardware devices and assets across multiple customers. It functions as a microservices infrastructure that allows the deployment of data collection and management services as independent containerized units for scaling. The platform includes a rule-based stream processor that transforms incoming device data and triggers alarms using customizable rule chains. It also provides a data visualization suite consisting of dashboards and widgets to display real-time telemetry and syste

Age is a command-line utility for file encryption that utilizes hybrid cryptography to secure data for multiple recipients. It employs a combination of asymmetric key exchange and symmetric encryption to protect files, supporting access control through public keys, shared passphrases, and hardware-backed identity integration. The tool is designed for memory-efficient operation, utilizing stream-oriented processing to handle large datasets in small, sequential chunks. It features a stanza-based metadata framing system that allows for extensible file headers and supports random-access decryptio

ioredis is a performance-focused Redis client for Node.js designed to execute commands and manage data connections. It provides a specialized interface for interacting with standalone servers, sharded clusters, and high-availability setups. The library distinguishes itself with native support for Redis Cluster, featuring automatic slot discovery and network address mapping, and Redis Sentinel for master node discovery and automatic failover. It also includes a dedicated Lua scripting interface that utilizes server-side caching to ensure atomic operations. The project covers a broad set of ca

ioredis is a performance-focused Redis client for Node.js applications. It provides a comprehensive interface for interacting with Redis servers, including specialized clients for sharded clusters and Sentinel-based high availability environments. The project distinguishes itself through advanced networking and execution capabilities, such as automatic event-loop pipelining to reduce overhead and a system for routing read-write traffic between primary and replica nodes. It also features a dedicated Lua scripting interface that allows server-side scripts to be registered as custom client comma

CryptoSwift is a cryptography library implemented entirely in the Swift programming language. It provides a collection of standard cryptographic algorithms for encryption, decryption, and hashing without relying on native C libraries or system frameworks. The library supports symmetric and asymmetric encryption, including RSA key generation and signature management. It features authenticated encryption schemes and the ability to generate cryptographic digests for data integrity verification. The toolset covers message authentication codes, secure key derivation from passwords, and data paddi

phpredis is a C-based native extension that bridges PHP applications with Redis servers for high-performance data storage and retrieval. It serves as an interface for manipulating strings, hashes, lists, sets, and sorted sets while providing a direct path for executing Redis commands and server-side scripts. The extension provides comprehensive support for distributed environments and high availability. It interfaces with Redis Cluster to distribute data across multiple nodes using hash slots and manages Redis Sentinel for service discovery and automatic failover. It also enables shared state

Redis is a high-performance in-memory key-value store that functions as a distributed cache, message broker, and NoSQL database. It provides sub-millisecond read and write access to data stored in RAM and can operate as a vector database for indexing high-dimensional embeddings. The system supports a wide range of data storage and synchronization primitives, including the management of strings, hashes, lists, sets, and JSON documents. It enables real-time data operations through atomic transactions, hybrid persistence using snapshots and append-only logs, and high-availability configurations

Watermill is a Go library for building event-driven applications. It provides a framework for implementing event-driven architectures through a consistent interface for message publishing and subscribing, featuring a pub/sub message router and event stream processing pipelines. The library uses interface-based abstractions to decouple message producers and consumers from the underlying message broker. This allows different messaging technologies to be swapped via plugin-based broker integration. It employs a router-driven dispatch system and middleware-based pipelines to manage cross-cutting

RedisInsight is a graphical user interface and management tool for browsing, analyzing, and administering Redis databases. It provides a visual environment for exploring key-value data structures, managing database instances, and performing data analysis across different operating systems and deployments. The tool distinguishes itself by providing dedicated visual managers for complex operations, including a vector database manager for configuring embeddings and similarity searches, a query workbench for executing raw commands and Lua scripts, and a performance monitoring dashboard for tracki

Enterprise job scheduling middleware with distributed computing ability.

Faust is a Python library for building distributed stream processing applications that integrate with Kafka. It functions as an asynchronous stream processor designed to handle high-throughput event streams and real-time data analysis using asynchronous functions. The system operates as a distributed stream processor and state store, utilizing sharding and partitioned topics to scale processing workloads horizontally across multiple worker nodes. It maintains state through a replicated key-value storage system backed by local databases to ensure high availability and fast recovery. The frame

Feast is an open-source feature store for machine learning that provides a central platform for defining, storing, and serving features across both training and inference workflows. It operates as a declarative system where feature definitions are written as code in Python files, synchronized to a central registry, and made available for low-latency online retrieval or point-in-time correct historical joins for training datasets. The project abstracts storage behind a pluggable architecture, allowing offline and online backends to be swapped without changing retrieval logic, and coordinates ma

Octosql este un motor de interogare SQL federat, un transformator de date și un procesor de SQL pentru streaming. Acesta permite utilizatorilor să execute instrucțiuni SQL unice peste surse de date multiple și disparate, inclusiv diferite tipuri de baze de date și formate de fișiere, pentru a îmbina și transforma rezultatele într-un set unitar. Sistemul se distinge prin tratarea fișierelor CSV, JSONLines și Parquet ca tabele virtuale și prin utilizarea unei arhitecturi bazate pe plugin-uri pentru a extinde conectivitatea către motoare de stocare externe. Funcționează ca un procesor de streaming pentru fluxuri de date infinite, folosind watermarks, retractions și tumbling windows pentru a menține consistența evenimentelor sosite în afara ordinii. În plus, servește ca generator de date SQL capabil să producă seturi de date sintetice și fluxuri de înregistrări prin funcții de tip tabel. Motorul include capabilități pentru join-uri cross-source și analiză multi-sursă, optimizate prin predicate push-down la nivel de sursă pentru a reduce transferul de date. Gestionează date complexe printr-un sistem de tipuri statice cu union types și oferă observabilitate prin vizualizarea planurilor de execuție a interogărilor.

Riemann este un procesor de fluxuri de evenimente și un motor de analiză în timp real bazat pe Clojure. Acesta funcționează ca o conductă de telemetrie de rețea și un router de evenimente extensibil care preia, transformă și direcționează datele de evenimente din sisteme distribuite. Sistemul utilizează un limbaj specific domeniului (DSL) pentru a calcula metrici și tipare statistice pe fluxuri continue, permițând analiza tendințelor de rețea și alertarea în timp real. Suportă încărcarea dinamică a plugin-urilor din classpath și permite reîncărcarea configurației în direct, fără a întrerupe fluxurile de evenimente active. Capabilitățile includ agregarea centralizată a telemetriei, etichetarea metadatelor evenimentelor și indexarea stării evenimentelor. Sistemul gestionează distribuția fluxurilor de evenimente prin divizare, procesare în loturi (batching) și filtrare, oferind în același timp transport securizat în rețea prin criptare și autentificare.

RealtimeTTS is a real-time text-to-speech engine and stream processor designed to convert text or token streams into audio playback with minimal latency. It provides a programmatic interface for managing audio streams, synthesis progress, and the integration of local or cloud-based speech engines. The system includes a neural voice cloning tool that generates synthetic speech by extracting acoustic features from reference audio samples. It utilizes a provider-based abstraction to route synthesis requests across different neural models and cloud APIs. The project covers a range of functional