apache/storm
Storm
Storm is a distributed stream processing framework designed to execute unbounded computations across a cluster to process real-time data streams. It functions as a data pipeline orchestrator that allows users to define and deploy declarative data flow graphs connecting streaming sources to processing components.
The system operates as a multi-tenant distributed compute engine that isolates workloads and limits resource usage across shared clusters using dedicated pools and access control. It is also a secure distributed processing engine that employs encrypted node communication and SSL-secured management interfaces to protect data stream execution.
The platform provides capabilities for external data system integration, connecting processing logic with external storage and messaging systems. It includes a suite of security and identity tools for user authentication, access control enforcement, and the secure management of distributed computing environments.
Features
- Streaming Data Processing - Executes unbounded computations across a distributed system to perform streaming data processing in real time.
- Real-Time Data Streaming - Executes unbounded computations on data streams as they arrive across a distributed system for low-latency results.
- Data Flow Definitions - Allows users to define data flow graphs and grouping strategies using a declarative definition.
- Secure Computing Frameworks - Protects cluster communication and management interfaces using encryption and authentication to secure data flows.
- Real-Time Data Processors - Orchestrates declarative data flow graphs that connect streaming sources to processing components.
- Secure Processing Engines - Implements a processing engine with encrypted node communication and SSL-secured management interfaces.
- External Data Ingestion - Connects to diverse streaming and storage systems for the ingestion of real-time data.
- External Data Integrations - Connects streaming processing logic with external storage and messaging systems for real-time data ingestion and persistence.
- Workload Isolation - Allocates dedicated resource pools to specific users to prevent cross-tenant interference in shared clusters.
- Stream Processing Pipeline Deployments - Provides capabilities for deploying fault-tolerant stream processing pipelines via declarative topologies.
- Message Passing - Uses tuple-based message passing to communicate standardized data records between processing components.
- Multi-Tenant Isolation - Isolates workloads and limits resource usage across shared clusters using dedicated pools and access control.
- Cluster Resource Isolation - Isolates workloads and limits resource usage for different users to ensure stability and fairness in shared clusters.
- Distributed Cluster Coordination - Utilizes Zookeeper for distributed cluster coordination, managing leader election and worker synchronization.
- Directed Acyclic Graph Pipelines - Implements data processing using directed acyclic graph pipelines to route tuples through interconnected spouts and bolts.
- External System Integrations - Provides mechanisms to bridge processing logic with external systems and tools.
- Resource Caps - Caps the maximum number of slots and executors a single process can utilize to maintain system stability.
- Authentication Plugins - Provides a modular system of authentication plugins to integrate external identity providers across the distributed cluster.
- Access Control and Authorization - Enforces access control and authorization by restricting operations to authorized users via access control lists.
- Identity Authentication - Verifies user identities across distributed components using secure authentication protocols and plugins.
- Secure Node Networking - Encrypts and authenticates messaging between worker nodes to prevent unauthorized data processing.
- Management Interface Security - Provides encrypted communication for user interfaces and RPC endpoints using SSL and mutual authentication.
- Master-Worker Process Models - Implements a master-worker process model where supervisor daemons monitor and restart failed worker processes.
- Process Isolation Architectures - Employs process isolation architectures by running processing tasks in separate JVM processes to ensure node stability.
- Resource Slot Scheduling - Uses resource slot scheduling to divide hardware into fixed execution slots, limiting concurrent tasks per node.
- Big Data Frameworks - Distributed real-time computation system.
- Distributed Applications - Distributed system for real-time stream processing.
- Streaming Engines - Distributed real-time computation system for unbounded streams.
- External Integrations - Integration for real-time distributed computation.
