Awesome GitHub RepositoriesStreaming Processors

Apache Flink is a distributed processing engine designed for both high-throughput, low-latency data streams and finite batch workloads. It functions as a stateful stream processor and a SQL stream processing engine, providing a unified runtime to execute relational queries and event-based transformations. The system is distinguished by its ability to manage persistent operator state to ensure exactly-once processing guarantees and consistency during failures. It features specialized capabilities for complex event processing to detect temporal patterns and handles out-of-order events using eve

Toon is a data serialization library and toolkit designed to convert complex objects into compact, human-readable formats optimized for large language models. By focusing on token efficiency, the library minimizes the context window footprint of structured data through techniques like key folding and tabular layout optimization. It provides a streaming-capable processor that handles the encoding and decoding of hierarchical data while maintaining structural integrity. The project distinguishes itself through its path-aware transformation pipeline and configurable serialization logic, which al

This project is a feature-rich Go client library designed for interacting with Redis. It serves as a comprehensive interface for managing remote data stores, enabling developers to execute standard database commands, handle complex data structures, and perform asynchronous operations within Go applications. The library distinguishes itself through its support for advanced Redis capabilities, including connection pooling, pipelining, and transactional integrity. It provides specialized primitives for managing distributed clusters, including automated topology updates and request routing to sha

ThingsBoard is an IoT device management platform designed for provisioning, monitoring, and managing large fleets of hardware devices and assets across multiple customers. It functions as a microservices infrastructure that allows the deployment of data collection and management services as independent containerized units for scaling. The platform includes a rule-based stream processor that transforms incoming device data and triggers alarms using customizable rule chains. It also provides a data visualization suite consisting of dashboards and widgets to display real-time telemetry and syste

Age is a command-line utility for file encryption that utilizes hybrid cryptography to secure data for multiple recipients. It employs a combination of asymmetric key exchange and symmetric encryption to protect files, supporting access control through public keys, shared passphrases, and hardware-backed identity integration. The tool is designed for memory-efficient operation, utilizing stream-oriented processing to handle large datasets in small, sequential chunks. It features a stanza-based metadata framing system that allows for extensible file headers and supports random-access decryptio

ioredis is a performance-focused Redis client for Node.js designed to execute commands and manage data connections. It provides a specialized interface for interacting with standalone servers, sharded clusters, and high-availability setups. The library distinguishes itself with native support for Redis Cluster, featuring automatic slot discovery and network address mapping, and Redis Sentinel for master node discovery and automatic failover. It also includes a dedicated Lua scripting interface that utilizes server-side caching to ensure atomic operations. The project covers a broad set of ca

ioredis is a performance-focused Redis client for Node.js applications. It provides a comprehensive interface for interacting with Redis servers, including specialized clients for sharded clusters and Sentinel-based high availability environments. The project distinguishes itself through advanced networking and execution capabilities, such as automatic event-loop pipelining to reduce overhead and a system for routing read-write traffic between primary and replica nodes. It also features a dedicated Lua scripting interface that allows server-side scripts to be registered as custom client comma

CryptoSwift is a cryptography library implemented entirely in the Swift programming language. It provides a collection of standard cryptographic algorithms for encryption, decryption, and hashing without relying on native C libraries or system frameworks. The library supports symmetric and asymmetric encryption, including RSA key generation and signature management. It features authenticated encryption schemes and the ability to generate cryptographic digests for data integrity verification. The toolset covers message authentication codes, secure key derivation from passwords, and data paddi

phpredis is a C-based native extension that bridges PHP applications with Redis servers for high-performance data storage and retrieval. It serves as an interface for manipulating strings, hashes, lists, sets, and sorted sets while providing a direct path for executing Redis commands and server-side scripts. The extension provides comprehensive support for distributed environments and high availability. It interfaces with Redis Cluster to distribute data across multiple nodes using hash slots and manages Redis Sentinel for service discovery and automatic failover. It also enables shared state

Redis is a high-performance in-memory key-value store that functions as a distributed cache, message broker, and NoSQL database. It provides sub-millisecond read and write access to data stored in RAM and can operate as a vector database for indexing high-dimensional embeddings. The system supports a wide range of data storage and synchronization primitives, including the management of strings, hashes, lists, sets, and JSON documents. It enables real-time data operations through atomic transactions, hybrid persistence using snapshots and append-only logs, and high-availability configurations

Watermill is a Go library for building event-driven applications. It provides a framework for implementing event-driven architectures through a consistent interface for message publishing and subscribing, featuring a pub/sub message router and event stream processing pipelines. The library uses interface-based abstractions to decouple message producers and consumers from the underlying message broker. This allows different messaging technologies to be swapped via plugin-based broker integration. It employs a router-driven dispatch system and middleware-based pipelines to manage cross-cutting

RedisInsight is a graphical user interface and management tool for browsing, analyzing, and administering Redis databases. It provides a visual environment for exploring key-value data structures, managing database instances, and performing data analysis across different operating systems and deployments. The tool distinguishes itself by providing dedicated visual managers for complex operations, including a vector database manager for configuring embeddings and similarity searches, a query workbench for executing raw commands and Lua scripts, and a performance monitoring dashboard for tracki

Enterprise job scheduling middleware with distributed computing ability.

Faust is a Python library for building distributed stream processing applications that integrate with Kafka. It functions as an asynchronous stream processor designed to handle high-throughput event streams and real-time data analysis using asynchronous functions. The system operates as a distributed stream processor and state store, utilizing sharding and partitioned topics to scale processing workloads horizontally across multiple worker nodes. It maintains state through a replicated key-value storage system backed by local databases to ensure high availability and fast recovery. The frame

Feast is an open-source feature store for machine learning that provides a central platform for defining, storing, and serving features across both training and inference workflows. It operates as a declarative system where feature definitions are written as code in Python files, synchronized to a central registry, and made available for low-latency online retrieval or point-in-time correct historical joins for training datasets. The project abstracts storage behind a pluggable architecture, allowing offline and online backends to be swapped without changing retrieval logic, and coordinates ma

Octosql es un motor de consultas SQL federado, transformador de datos y procesador de SQL en streaming. Permite a los usuarios ejecutar sentencias SQL únicas a través de múltiples fuentes de datos dispares, incluyendo diferentes tipos de bases de datos y formatos de archivo, para combinar y transformar resultados en un conjunto unificado. El sistema se distingue por tratar archivos CSV, JSONLines y Parquet como tablas virtuales y utilizar una arquitectura basada en plugins para extender la conectividad a motores de almacenamiento externos. Funciona como un procesador de streaming para flujos de datos infinitos, utilizando marcas de agua (watermarks), retracciones y ventanas deslizantes (tumbling windows) para mantener la consistencia en eventos fuera de orden. Además, sirve como generador de datos SQL capaz de producir conjuntos de datos sintéticos y flujos de registros mediante funciones con valores de tabla. El motor incluye capacidades para realizar joins entre fuentes de datos y análisis multi-fuente, optimizado mediante el push-down de predicados en el lado de la fuente para reducir la transferencia de datos. Gestiona datos complejos a través de un sistema de tipos estáticos con tipos unión y proporciona observabilidad mediante la visualización de planes de ejecución de consultas.

Riemann is a Clojure-based event stream processor and real-time analytics engine. It functions as a network telemetry pipeline and extensible event router that ingests, transforms, and routes event data from distributed systems. The system uses a domain-specific language to compute metrics and statistical patterns over continuous streams, enabling network trend analysis and real-time alerting. It supports dynamic plugin loading from the classpath and allows for live configuration reloading without interrupting active event streams. Capabilities include centralized telemetry aggregation, even

RealtimeTTS es un motor de texto a voz (TTS) en tiempo real y procesador de flujos diseñado para convertir texto o flujos de tokens en reproducción de audio con una latencia mínima. Proporciona una interfaz programática para gestionar flujos de audio, el progreso de la síntesis y la integración de motores de voz locales o basados en la nube. El sistema incluye una herramienta de clonación de voz neuronal que genera voz sintética extrayendo características acústicas de muestras de audio de referencia. Utiliza una abstracción basada en proveedores para enrutar las solicitudes de síntesis a través de diferentes modelos neuronales y APIs en la nube. El proyecto cubre una serie de dominios funcionales que incluyen control de reproducción asíncrona, síntesis de audio a partir de flujos de tokens y flujos de trabajo de sincronización de voz que activan callbacks a nivel de palabra o carácter. También admite la exportación de audio sintetizado directamente a archivos WAV para almacenamiento persistente.