Projects sharing features with Incubator Storm

Fluvio is a distributed event streaming platform and cloud-native streaming engine designed for collecting, persisting, and replicating real-time data streams across a distributed cluster. It functions as a real-time data pipeline for building stateful workflows that ingest, enrich, and export data between external sources and sinks. The platform is distinguished by its use of WebAssembly to execute compiled modules for in-line data transformations and filtering. This allows for the execution of custom business logic to reshape information in motion without requiring a restart of the cluster.

Apache Spark is a unified distributed data processing engine designed for large-scale data analysis and computation graphs. It functions as a distributed machine learning framework, a graph processing system, a real-time stream processor, and a SQL analytics engine. The system enables the execution of distributed SQL querying, large-scale graph analysis, and real-time stream analytics across clusters of machines. It also provides a scalable environment for implementing machine learning algorithms and predictive model development on massive datasets. The engine incorporates relational query e

jStorm is a distributed stream processing engine designed for executing low-latency computations on high-volume data streams using Apache Storm topologies. It functions as a real-time data analytics platform and distributed task orchestrator that manages complex data pipelines via directed acyclic graph execution. The system provides a scalable framework for data pipeline management, incorporating backpressure-aware flow control to regulate ingestion rates and dynamic resource allocation to adjust computing resources based on real-time demand. It maintains compatibility with Apache Storm conf

Pathway is a high-performance data processing framework designed for building unified batch and streaming pipelines. It functions as an orchestrator for complex data transformations, utilizing a differential dataflow engine to process updates incrementally. By treating static datasets and continuous event streams with identical logic, the platform ensures exactly-once processing semantics and consistent results across diverse data sources. The framework distinguishes itself through its specialized support for real-time artificial intelligence and retrieval-augmented generation. It features in

Arroyo is a high-performance stream processing platform built in Rust. It executes continuous SQL queries on streaming data with event-time semantics, enabling accurate windowed aggregations, joins, and stateful computations on unbounded event streams. The platform uses native Rust execution for high throughput and low latency, with periodic checkpointing for exactly-once fault tolerance and horizontal scaling across distributed workers. The system integrates deeply with Kafka for reading and writing topics with exactly-once delivery and supports change data capture (CDC) from MySQL and Postg

Storm is a distributed stream processing framework and fault-tolerant compute engine designed for executing real-time continuous computations across a cluster of machines. It functions as a stateful stream processor and cluster topology manager, enabling the deployment and monitoring of distributed data flow configurations. The system ensures exactly-once semantics by utilizing transactional state management to guarantee that every message in a data stream is processed exactly one time. It further operates as a distributed RPC system, allowing for the integration of non-native languages throu

Light Task Scheduler is a distributed job scheduling and workflow orchestration platform designed for managing background processing across scalable computing environments. It functions as a cluster management system that coordinates stateless nodes to execute recurring, cron-based, or one-time tasks with centralized control and high availability. The platform distinguishes itself through a leader-based coordination model that automatically elects a primary controller to manage task distribution and system state. It supports complex workflow dependencies, ensuring that prerequisite tasks comp

nom is a parser combinator framework for Rust used to build complex parsers by combining small, reusable parsing functions. It functions as a zero-copy parsing tool that minimizes memory overhead by returning slices of the original input instead of allocating new memory. The framework is designed for diverse data formats, serving as a binary data parser with configurable endianness and a bitstream processing library capable of extracting values of arbitrary bit length. It also functions as a streaming data parser that can process data arriving in chunks and signal when additional input is req

Apache Beam is a distributed data pipeline framework and unified data processing model designed to handle both bounded batch data and unbounded real-time streams. It provides a system for building scalable, data-parallel workflows that operate across compute clusters using a single programming model. The framework utilizes a cross-runner pipeline abstraction that decouples the data processing logic from the underlying execution backend, allowing the same pipeline to run on different distributed compute engines. It supports multi-language pipeline development by translating high-level code fro

Orleans is a .NET distributed actor framework designed for building scalable, cloud-native applications. It implements a virtual actor model where entities with stable identities manage their own state and lifecycle across a cluster of servers. The framework provides a distributed state management system with ACID transaction support and a distributed pub/sub streaming engine for real-time data processing. It distinguishes itself through location-transparent routing, automatic actor activation and deactivation, and elastic cluster scaling that redistributes workloads during node failures. Th

This project is a Python wrapper for the TA-Lib library, providing a technical analysis library for computing moving averages, momentum, and volatility metrics for financial time series analysis. It serves as a financial indicator calculator that processes price and volume arrays to generate technical signals and pattern recognition. The library includes an incremental data processor capable of computing the most recent technical indicator values as new streaming market data arrives. This allows for real-time price monitoring and the processing of streaming data without recalculating entire d

franz-go is a low-level Go client library and wire protocol implementation for producing, consuming, and administering Kafka clusters. It functions as a zero-allocation network driver that utilizes a direct TCP communication layer to handle requests and responses. The project integrates a schema registry client for encoding and decoding structured data. It provides a programmatic interface for cluster administration, including the management of topics, access control lists, and broker configurations. The library covers data consumption through consumer groups, message production with transac

Resilience4j is a fault tolerance library for Java 8 applications designed to handle failures in distributed systems. It provides a comprehensive suite of resilience patterns including resource isolation, request throttling, retries, and timeouts. The project implements specific stability mechanisms such as circuit breakers to prevent cascading failures, bulkheads for system resource isolation, and rate limiters to cap execution rates. It also includes a retry framework with backoff policies and timeout enforcement to manage transient faults and maximum execution durations. Additional capabi

nom is a Rust parser combinator framework used to build complex parsers for binary and text data. It functions as an abstract syntax tree generator and a bit-level binary parser, allowing users to construct structured data by combining small, reusable parsing functions. The framework provides specialized support for zero-copy binary parsing, extracting data as slices from raw byte arrays to avoid memory allocations. It also includes a streaming data parser capable of processing partial input chunks from networks or files and signaling when additional input is required. The project covers a b

YouPlot is a command line plotting utility and terminal data visualization tool used to render statistical plots and charts directly within a terminal interface using Unicode characters. It functions as a Unix pipeline plotter, allowing users to visualize numerical data without leaving the shell. The project operates as a real-time data visualizer, drawing plots progressively as data streams into the system. It integrates into command line pipelines by reading data from standard input to provide real-time stream monitoring and data analysis. The tool covers a variety of rendering capabilitie

Polly is a resilience and transient-fault-handling library for .NET applications. It provides a framework for defining and executing stability patterns that manage failures and resource exhaustion during operation execution. The library enables developers to compose complex resilience strategies through a fluent builder pattern, wrapping target operations in functional decorators. These policies intercept execution flow to apply automated recovery mechanisms, such as retries for temporary errors or circuit breakers that isolate unstable components by blocking requests. The framework supports

Dora is a robotics dataflow framework and distributed orchestrator used to build and manage processing pipelines. It enables the deployment of robotics workloads across clusters with remote node execution and provides a real-time data pipeline for predictable performance. The system is distinguished by its support for multi-language nodes written in Rust, Python, C, or C++ that interoperate within a single dataflow. It utilizes a zero-copy shared-memory transport and columnar formats to minimize latency for large payloads, and it includes bidirectional bridges to integrate with external ecosy

SpringCloud-Learning is an educational project that demonstrates how to build microservices using Spring Cloud, covering the core patterns of service discovery, centralized configuration management, and API gateway routing. The project provides hands-on examples for registering and discovering microservice instances with Nacos, Eureka, or Consul, and for routing external API requests through Spring Cloud Gateway with support for filters and load balancing. The tutorials explore service resilience through circuit breakers and rate limiting with Sentinel and Hystrix, including custom fallback l

litegraph.js is a JavaScript dataflow framework and visual node graph engine used to define programmable logic and data flow. It provides a node-based visual programming tool for designing complex logic through connected functional blocks. The library allows for the creation of hierarchical logic by nesting multiple nodes into recursive subgraphs. It also supports the development of custom node types with unique inputs and outputs, as well as custom widgets and live views that can hide the underlying graph structure to present a visual interface. The engine enables the execution of logic gra

SeaTunnel is a distributed data integration engine designed to synchronize structured and unstructured data across diverse sources and sinks. It functions as a multi-engine execution framework that can run data integration tasks across different distributed computing backends to optimize workload performance. The project is distinguished by a visual data pipeline designer for configuring workflows without manual code and a specialized change data capture tool for streaming incremental database updates. It also includes an enrichment pipeline that integrates large language models and embedding

Sarama is an Apache Kafka Go client library that provides native support for the Kafka protocol. It includes a protocol client for managing offsets and timestamps, a producer implementation for sending messages, and a consumer group coordinator to balance workloads across multiple instances. The library enables high throughput data streaming through concurrent message production and maintains strict partition ordering during network retries. It supports secure communication with Kafka brokers using certificate-based encryption to protect data traffic. The project covers a broad range of dist

GreptimeDB is a distributed, open-source time-series database built for unified observability. It stores and queries metrics, logs, and traces together in a single columnar engine, supporting both SQL and PromQL for analysis. The database is designed as a Kubernetes-native operator with a decoupled compute and storage architecture, enabling horizontal scaling and multi-region deployment. What distinguishes GreptimeDB is its role as a multi-protocol ingestion gateway, accepting data through OpenTelemetry, Prometheus Remote Write, InfluxDB, Loki, Elasticsearch, Kafka, and MQTT protocols without

Failsafe is a JVM resilience library providing a collection of fault tolerance patterns for Java applications. It functions as an asynchronous execution wrapper that runs tasks in the background and returns futures to prevent thread blocking. The library allows for the composition of resilience policies, enabling multiple patterns to be stacked into a sequential pipeline. It includes specific implementations for circuit breaking to prevent system overload, rate limiting to control traffic flow, and a framework for managing retries and fallbacks. Capability areas cover traffic management thro

This project is a comprehensive Java backend engineering guide and technical reference focused on high-concurrency design, distributed systems, and microservices architecture. It provides detailed strategies for decomposing monolithic applications, managing service discovery, and implementing the architectural patterns required for scalable backend environments. The repository distinguishes itself through an extensive collection of big data algorithmic references and database scaling strategies. It covers memory-efficient techniques for analyzing massive datasets, such as Top-K element extrac

Akka is an actor model framework and distributed systems platform used to build concurrent and distributed applications. It provides a toolkit for managing multi-threaded state and behavior through asynchronous message passing, allowing developers to create concurrent applications without manual locks or synchronization. The system functions as a cluster management and event sourcing framework, automating the scaling and coordination of high-availability clusters. It enables the deployment of elastic services that coordinate workloads across multiple network nodes and ensures fault tolerance

RocketMQ is a distributed messaging and streaming platform designed for building event-driven applications. It serves as middleware to decouple services using publish-subscribe and request-reply patterns, and functions as a transactional messaging system that ensures atomicity by linking message delivery to local transaction outcomes. The platform includes specialized capabilities as a Kubernetes-native message broker for container orchestration environments and an MQTT broker for ingesting event data from mobile applications and hardware terminals. The system covers high-throughput data str

Perspective is a columnar data analytics library and streaming data visualization engine. It provides an interactive data grid component and notebook analytics widgets designed for processing high-volume data and rendering interactive charts and grids. The system utilizes a high-performance query engine to enable real-time data analysis and streaming dataset visualization. It supports the creation of customizable dashboards and reports that update automatically as new data arrives without requiring full dataset reloads. The project covers large-scale dataset analytics through a schema-driven