Awesome GitHub RepositoriesStreaming Processors

Apache Flink is a distributed processing engine designed for both high-throughput, low-latency data streams and finite batch workloads. It functions as a stateful stream processor and a SQL stream processing engine, providing a unified runtime to execute relational queries and event-based transformations. The system is distinguished by its ability to manage persistent operator state to ensure exactly-once processing guarantees and consistency during failures. It features specialized capabilities for complex event processing to detect temporal patterns and handles out-of-order events using eve

Toon is a data serialization library and toolkit designed to convert complex objects into compact, human-readable formats optimized for large language models. By focusing on token efficiency, the library minimizes the context window footprint of structured data through techniques like key folding and tabular layout optimization. It provides a streaming-capable processor that handles the encoding and decoding of hierarchical data while maintaining structural integrity. The project distinguishes itself through its path-aware transformation pipeline and configurable serialization logic, which al

This project is a feature-rich Go client library designed for interacting with Redis. It serves as a comprehensive interface for managing remote data stores, enabling developers to execute standard database commands, handle complex data structures, and perform asynchronous operations within Go applications. The library distinguishes itself through its support for advanced Redis capabilities, including connection pooling, pipelining, and transactional integrity. It provides specialized primitives for managing distributed clusters, including automated topology updates and request routing to sha

ThingsBoard is an IoT device management platform designed for provisioning, monitoring, and managing large fleets of hardware devices and assets across multiple customers. It functions as a microservices infrastructure that allows the deployment of data collection and management services as independent containerized units for scaling. The platform includes a rule-based stream processor that transforms incoming device data and triggers alarms using customizable rule chains. It also provides a data visualization suite consisting of dashboards and widgets to display real-time telemetry and syste

Age is a command-line utility for file encryption that utilizes hybrid cryptography to secure data for multiple recipients. It employs a combination of asymmetric key exchange and symmetric encryption to protect files, supporting access control through public keys, shared passphrases, and hardware-backed identity integration. The tool is designed for memory-efficient operation, utilizing stream-oriented processing to handle large datasets in small, sequential chunks. It features a stanza-based metadata framing system that allows for extensible file headers and supports random-access decryptio

ioredis is a performance-focused Redis client for Node.js designed to execute commands and manage data connections. It provides a specialized interface for interacting with standalone servers, sharded clusters, and high-availability setups. The library distinguishes itself with native support for Redis Cluster, featuring automatic slot discovery and network address mapping, and Redis Sentinel for master node discovery and automatic failover. It also includes a dedicated Lua scripting interface that utilizes server-side caching to ensure atomic operations. The project covers a broad set of ca

ioredis is a performance-focused Redis client for Node.js applications. It provides a comprehensive interface for interacting with Redis servers, including specialized clients for sharded clusters and Sentinel-based high availability environments. The project distinguishes itself through advanced networking and execution capabilities, such as automatic event-loop pipelining to reduce overhead and a system for routing read-write traffic between primary and replica nodes. It also features a dedicated Lua scripting interface that allows server-side scripts to be registered as custom client comma

CryptoSwift is a cryptography library implemented entirely in the Swift programming language. It provides a collection of standard cryptographic algorithms for encryption, decryption, and hashing without relying on native C libraries or system frameworks. The library supports symmetric and asymmetric encryption, including RSA key generation and signature management. It features authenticated encryption schemes and the ability to generate cryptographic digests for data integrity verification. The toolset covers message authentication codes, secure key derivation from passwords, and data paddi

phpredis is a C-based native extension that bridges PHP applications with Redis servers for high-performance data storage and retrieval. It serves as an interface for manipulating strings, hashes, lists, sets, and sorted sets while providing a direct path for executing Redis commands and server-side scripts. The extension provides comprehensive support for distributed environments and high availability. It interfaces with Redis Cluster to distribute data across multiple nodes using hash slots and manages Redis Sentinel for service discovery and automatic failover. It also enables shared state

Redis is a high-performance in-memory key-value store that functions as a distributed cache, message broker, and NoSQL database. It provides sub-millisecond read and write access to data stored in RAM and can operate as a vector database for indexing high-dimensional embeddings. The system supports a wide range of data storage and synchronization primitives, including the management of strings, hashes, lists, sets, and JSON documents. It enables real-time data operations through atomic transactions, hybrid persistence using snapshots and append-only logs, and high-availability configurations

Watermill is a Go library for building event-driven applications. It provides a framework for implementing event-driven architectures through a consistent interface for message publishing and subscribing, featuring a pub/sub message router and event stream processing pipelines. The library uses interface-based abstractions to decouple message producers and consumers from the underlying message broker. This allows different messaging technologies to be swapped via plugin-based broker integration. It employs a router-driven dispatch system and middleware-based pipelines to manage cross-cutting

RedisInsight is a graphical user interface and management tool for browsing, analyzing, and administering Redis databases. It provides a visual environment for exploring key-value data structures, managing database instances, and performing data analysis across different operating systems and deployments. The tool distinguishes itself by providing dedicated visual managers for complex operations, including a vector database manager for configuring embeddings and similarity searches, a query workbench for executing raw commands and Lua scripts, and a performance monitoring dashboard for tracki

Enterprise job scheduling middleware with distributed computing ability.

Faust is a Python library for building distributed stream processing applications that integrate with Kafka. It functions as an asynchronous stream processor designed to handle high-throughput event streams and real-time data analysis using asynchronous functions. The system operates as a distributed stream processor and state store, utilizing sharding and partitioned topics to scale processing workloads horizontally across multiple worker nodes. It maintains state through a replicated key-value storage system backed by local databases to ensure high availability and fast recovery. The frame

Feast is an open-source feature store for machine learning that provides a central platform for defining, storing, and serving features across both training and inference workflows. It operates as a declarative system where feature definitions are written as code in Python files, synchronized to a central registry, and made available for low-latency online retrieval or point-in-time correct historical joins for training datasets. The project abstracts storage behind a pluggable architecture, allowing offline and online backends to be swapped without changing retrieval logic, and coordinates ma

Octosql هو محرك استعلامات SQL موزع، ومحول بيانات، ومعالج SQL للبث المباشر. يتيح للمستخدمين تنفيذ استعلامات SQL واحدة عبر مصادر بيانات متعددة ومتباينة، بما في ذلك أنواع قواعد البيانات المختلفة وتنسيقات الملفات، لدمج النتائج وتحويلها إلى مجموعة بيانات موحدة. يتميز النظام بمعاملة ملفات CSV وJSONLines وParquet كجداول افتراضية، ويستخدم بنية تعتمد على الإضافات (plugins) لتوسيع الاتصال بمحركات التخزين الخارجية. يعمل كمعالج للبث المباشر لتدفقات البيانات غير المحدودة، مستخدماً العلامات المائية (watermarks) وعمليات التراجع (retractions) والنوافذ الزمنية (tumbling windows) للحفاظ على الاتساق في الأحداث غير المرتبة. بالإضافة إلى ذلك، يعمل كمولد بيانات SQL قادر على إنتاج مجموعات بيانات اصطناعية وتدفقات سجلات عبر دوال ذات قيم جدولية. يتضمن المحرك قدرات لربط البيانات عبر مصادر متعددة والتحليل متعدد المصادر، مع تحسين الأداء عبر دفع التنبؤات (predicate push-down) إلى جانب المصدر لتقليل نقل البيانات. يدير النظام البيانات المعقدة من خلال نظام أنواع ثابت (static type system) مع أنواع اتحادية (union types) ويوفر إمكانية المراقبة عبر تصور خطط تنفيذ الاستعلامات.

Riemann is a Clojure-based event stream processor and real-time analytics engine. It functions as a network telemetry pipeline and extensible event router that ingests, transforms, and routes event data from distributed systems. The system uses a domain-specific language to compute metrics and statistical patterns over continuous streams, enabling network trend analysis and real-time alerting. It supports dynamic plugin loading from the classpath and allows for live configuration reloading without interrupting active event streams. Capabilities include centralized telemetry aggregation, even

RealtimeTTS هو محرك تحويل النص إلى كلام (TTS) في الوقت الفعلي ومعالج تدفق مصمم لتحويل تدفقات النص أو الرموز إلى تشغيل صوتي بأقل زمن انتقال. يوفر واجهة برمجية لإدارة تدفقات الصوت، وتقدم التوليف، وتكامل محركات الكلام المحلية أو السحابية. يتضمن النظام أداة لاستنساخ الصوت العصبي تولد كلاماً اصطناعياً عن طريق استخراج الميزات الصوتية من عينات صوتية مرجعية. ويستخدم تجريداً قائماً على الموفر لتوجيه طلبات التوليف عبر نماذج عصبية مختلفة وواجهات برمجة تطبيقات سحابية. يغطي المشروع مجموعة من المجالات الوظيفية بما في ذلك التحكم في التشغيل غير المتزامن، وتوليف الصوت من تدفق الرموز، وسير عمل مزامنة الكلام التي تطلق استدعاءات (callbacks) على مستوى الكلمة أو الحرف. كما يدعم تصدير الصوت المُولّد مباشرة إلى ملفات WAV للتخزين الدائم.