Storm

Storm is a distributed stream processing framework and fault-tolerant compute engine designed for executing real-time continuous computations across a cluster of machines. It functions as a stateful stream processor and cluster topology manager, enabling the deployment and monitoring of distributed data flow configurations.

The system ensures exactly-once semantics by utilizing transactional state management to guarantee that every message in a data stream is processed exactly one time. It further operates as a distributed RPC system, allowing for the integration of non-native languages through a standardized communication protocol.

The framework covers a broad range of capabilities including distributed stateful computation, cluster resource management, and the execution of system-level shell commands. It provides tools for monitoring stream performance, validating topology submissions, and implementing customizable data routing and serialization.

Features

Real-Time Data Streaming - Provides a platform for processing and delivering continuous data streams in real-time across a cluster.

Stream Topology Management - Provides a management system for deploying, rebalancing, and monitoring the lifecycle of distributed data flow configurations.

Exactly-Once Processing Semantics - Guarantees that every message in a data stream is processed exactly once despite system failures.

Distributed Processing Frameworks - Provides a framework for executing real-time continuous computations across a cluster of machines with low latency.

Distributed Computing - Executes large-scale stateful computations and real-time distributed queries across a cluster of machines.

Streaming State Management - Combines high-volume stream processing with distributed queries to maintain and retrieve real-time state.

Stateful Processing Backends - Maintains real-time state and distributed queries to enable stateful stream processing.

Exactly-Once Processing Guarantees - Ensures exactly-once semantics using transactional state management to guarantee reliable message processing.

Cross-Language RPC Frameworks - Implements a distributed RPC system that enables integration of non-native languages through a standardized communication protocol.

Distributed RPC Systems - Operates as a distributed RPC system for executing remote procedure calls and integrating external languages.

Fault Tolerance - Maintains transactional state and ensures resilience across distributed worker nodes to prevent data loss.

Clustered Task Distribution - Distributes programmatic data processing tasks across a cluster of machines using a coordinator.

Directed Acyclic Graph Pipelines - Defines data flow topologies as directed acyclic graphs to manage task dependencies and processing sequences.

Transactional State Tracking - Ensures exactly-once processing by tracking message offsets and committing state changes across the distributed pipeline.

Pluggable Serializers - Allows definition of custom serialization and deserialization logic using pluggable factories.

Stream Routing - Provides logic for directing real-time data streams across tasks using pluggable grouping strategies.

Machine-to-Workload Assignments - Assigns specific physical hardware to certain data flows to ensure complete resource isolation.

Cross-Language Runtime Integration - Integrates runtimes from different programming languages to allow the execution of non-native components.

Language Bridges - Provides interfaces for translating data types and function calls to execute non-native language components.

Stream Processing Resource Managers - Provides mechanisms to assign specific hardware to data flows and monitor throughput for resource isolation.

Distributed Cluster Coordination - Implements centralized synchronization and scheduling of data flows across compute nodes.

Stream Element Grouping - Routes data packets to downstream tasks using customizable key-extraction and grouping logic.

Stream Performance Monitoring - Tracks execution latency and data throughput to visualize system capacity and bottlenecks.

Topology Lifecycle Management - Provides tools to control the activation, rebalancing, and termination of running data flows.

Data Processing and Analytics - Distributed system for real-time stream processing.

nathanmarzstorm

Features

Star history