apache/incubator-storm
Incubator Storm
Apache Storm is a distributed stream processing framework and real-time data processing engine. It functions as a fault-tolerant distributed computing system designed to analyze data in motion across a cluster of machines for continuous stream computation.
The system enables the creation of fault-tolerant data pipelines and scalable event processing by distributing workloads across a network of computing nodes. This architecture ensures low latency and high throughput for live data while allowing the system to recover automatically from individual node failures.
The framework provides capabilities for distributed stream computing and the processing of real-time data streams to build scalable streaming applications.
Features
- Streaming Data Processing - Analyzes and transforms continuous streams of real-time data using a distributed computing framework.
- Stream Processing Engines - Provides a distributed engine for continuous computation on real-time data streams with low latency.
- Real-Time Data Processors - Ingests and transforms data streams in real-time for continuous analytics and event handling.
- Distributed Stream Execution - Executes streaming pipelines as a directed acyclic graph distributed across a cluster of worker nodes.
- High-Throughput Data Streaming - Provides a high-throughput architecture optimized for the continuous flow and processing of massive event volumes.
- Stream Processing Pipelines - Passes discrete data records through asynchronous message streams using a high-throughput pipeline architecture.
- Fault Tolerance - Ensures system resilience by automatically recovering from individual node failures in a distributed environment.
- Distributed Event Processing - Handles large volumes of events in motion by distributing workloads across a network of computing nodes.
- High-Performance Transport Layers - Utilizes a high-performance networking layer to move data tuples between nodes with minimal overhead.
- Reliable Transmission Mechanisms - Implements reliable data delivery through tuple tracking and automatic re-transmission of unacknowledged messages.
- Distributed Cluster Coordination - Uses ZooKeeper to synchronize cluster state and track the health of supervisor and worker nodes.
- Process Isolation Architectures - Runs processing tasks in separate JVM instances to ensure fault isolation and resource management.
- Databases & Data Processing - Distributed real-time computation system.
