apache/streampark
Streampark
StreamPark is a centralized management platform designed to coordinate the deployment, monitoring, and operational lifecycle of distributed stream processing and batch applications. It functions as a control plane and orchestrator for data pipelines, specifically providing management capabilities for Apache Flink and Hadoop YARN environments.
The platform distinguishes itself through a low-code approach to task deployment and a multi-engine execution adapter that supports diverse processing runtimes. It facilitates real-time data pipeline management by combining streaming SQL analytics with a resource-based deployment pipeline that handles versioning, binary uploads, and savepoint-based state recovery.
The system covers a broad set of capabilities including distributed job orchestration, real-time data integration via pre-built connectors, and identity integration through LDAP or SSO. It also provides observability tools for second-level application monitoring and automated operational fault notifications.
Features
- Distributed Job Orchestration - Provides a centralized framework for linking dependent tasks and managing the full lifecycle of distributed jobs.
- Apache Flink Management Platforms - Provides a centralized control plane for deploying, monitoring, and managing Apache Flink applications.
- Execution Engine Translation - Translates high-level task definitions into engine-specific configurations via an execution engine translation layer.
- Unified Batch and Stream Processing Engines - Executes both real-time streaming and batch workloads across different versions of processing engines.
- Low-Code Pipeline Managers - Offers a low-code platform for integrating data sources, executing streaming SQL, and managing warehouse connections.
- Stream and Pipeline Orchestration - Coordinates the flow, transformation, and distributed processing of continuous data streams across multiple engine versions.
- Distributed Task Schedulers - Orchestrates and distributes complex data processing workflows across computing clusters.
- Real-Time Data Integration Platforms - Synchronizes live data across heterogeneous storage and analytical environments using pre-built connectors.
- Streaming State Recovery - Recovers incremental operator states and output results from durable storage after failures.
- Unified Web Interfaces - Provides a unified web interface for building, deploying, and operating streaming applications across various engines.
- Centralized Application Maintenance Platforms - Coordinates the debugging, deployment, and maintenance of streaming applications through a centralized management platform.
- Hub-and-Spoke Control Planes - Implements a centralized control plane to manage the deployment and monitoring of batch and stream tasks.
- Stream Processing Coordinators - Coordinates the deployment and lifecycle of streaming and batch applications from a single centralized control point.
- Centralized Control Planes - Provides a centralized control interface for deploying and operating real-time data pipelines across multiple engines.
- Application Lifecycle Management - Manages the operational state, version updates, and health of running distributed stream processing tasks.
- YARN and Cluster Management - Manages resources and executes streaming jobs within Hadoop YARN clusters.
- Data Connectors - Utilizes pre-built connectors to efficiently acquire and move data between various sources and destinations.
- Development Scaffolding - Provides standardized configurations and development scaffolds to implement streaming logic across multiple processing engines.
- Data Processing Tasks - Handles compute-intensive operations including startup, savepoints, and performance analysis of stream jobs.
- Data Warehouse Integrations - Links processing engines with open-source data lakes and real-time warehouses to streamline storage and retrieval.
- Hadoop Workflow Orchestrators - Schedules and monitors big data workloads running on Hadoop clusters via HDFS.
- Real-Time Data Streaming - Executes interactive streaming queries and manages streaming warehouses for real-time data analysis.
- SQL Statement Executions - Provides capabilities to register tables and execute SQL statements for interfacing with external systems.
- Streaming SQL Transformations - Executes SQL queries directly against live data streams for real-time filtering and aggregation.
- Stream Processing Scaffolds - Provides development scaffolds and connectors to create stream processing applications compatible with multiple engine versions.
- Job Resource Uploaders - Handles the uploading of binaries and configuration dependencies to distributed filesystems prior to job submission.
- Low-Code Automation Platforms - Provides a low-code interface to coordinate the compilation, publishing, and deployment of processing tasks.
- Task Release Tracking - Tracks custom files and task releases to facilitate coordinated updates and rollbacks across multiple versions.
- Task Lifecycle Management - Governs the execution flow of tasks through explicit stages from initialization to destruction.
- Metadata-Driven Orchestration - Uses an external database for metadata-driven orchestration of distributed processing job states and configurations.
- Real-Time Application Performance Monitors - Provides live analysis of application performance metrics and health with second-level monitoring.
- Stream Performance Monitoring - Tracks latency, throughput, and health of distributed data processing pipelines with automated alerts.
