apache/streampark
Streampark
StreamPark ist eine zentralisierte Managementplattform, die darauf ausgelegt ist, das Deployment, Monitoring und den operativen Lebenszyklus verteilter Stream-Processing- und Batch-Anwendungen zu koordinieren. Sie fungiert als Control-Plane und Orchestrator für Datenpipelines und bietet spezifisch Managementfunktionen für Apache Flink- und Hadoop YARN-Umgebungen.
Die Plattform zeichnet sich durch einen Low-Code-Ansatz für das Task-Deployment und einen Multi-Engine-Execution-Adapter aus, der diverse Verarbeitungs-Runtimes unterstützt. Sie erleichtert das Echtzeit-Datenpipeline-Management durch die Kombination von Streaming-SQL-Analytics mit einer ressourcenbasierten Deployment-Pipeline, die Versionierung, Binär-Uploads und Savepoint-basierte Zustands-Wiederherstellung handhabt.
Das System deckt ein breites Spektrum an Funktionen ab, einschließlich verteilter Job-Orchestrierung, Echtzeit-Datenintegration über vorgefertigte Connectors und Identitätsintegration via LDAP oder SSO. Es bietet zudem Observability-Tools für sekundengenaue Anwendungsüberwachung und automatisierte operative Fehlerbenachrichtigungen.
Features
- Distributed Job Orchestration - Provides a centralized framework for linking dependent tasks and managing the full lifecycle of distributed jobs.
- Apache Flink Management Platforms - Provides a centralized control plane for deploying, monitoring, and managing Apache Flink applications.
- Execution Engine Translation - Translates high-level task definitions into engine-specific configurations via an execution engine translation layer.
- Unified Batch and Stream Processing Engines - Executes both real-time streaming and batch workloads across different versions of processing engines.
- Low-Code Pipeline Managers - Offers a low-code platform for integrating data sources, executing streaming SQL, and managing warehouse connections.
- Stream and Pipeline Orchestration - Coordinates the flow, transformation, and distributed processing of continuous data streams across multiple engine versions.
- Distributed Task Schedulers - Orchestrates and distributes complex data processing workflows across computing clusters.
- Real-Time Data Integration Platforms - Synchronizes live data across heterogeneous storage and analytical environments using pre-built connectors.
- Streaming State Recovery - Recovers incremental operator states and output results from durable storage after failures.
- Unified Web Interfaces - Provides a unified web interface for building, deploying, and operating streaming applications across various engines.
- Centralized Application Maintenance Platforms - Coordinates the debugging, deployment, and maintenance of streaming applications through a centralized management platform.
- Hub-and-Spoke Control Planes - Implements a centralized control plane to manage the deployment and monitoring of batch and stream tasks.
- Stream Processing Coordinators - Coordinates the deployment and lifecycle of streaming and batch applications from a single centralized control point.
- Centralized Control Planes - Provides a centralized control interface for deploying and operating real-time data pipelines across multiple engines.
- Application Lifecycle Management - Manages the operational state, version updates, and health of running distributed stream processing tasks.
- YARN and Cluster Management - Manages resources and executes streaming jobs within Hadoop YARN clusters.
- Data Connectors - Utilizes pre-built connectors to efficiently acquire and move data between various sources and destinations.
- Development Scaffolding - Provides standardized configurations and development scaffolds to implement streaming logic across multiple processing engines.
- Data Processing Tasks - Handles compute-intensive operations including startup, savepoints, and performance analysis of stream jobs.
- Data Warehouse Integrations - Links processing engines with open-source data lakes and real-time warehouses to streamline storage and retrieval.
- Hadoop Workflow Orchestrators - Schedules and monitors big data workloads running on Hadoop clusters via HDFS.
- Real-Time Data Streaming - Executes interactive streaming queries and manages streaming warehouses for real-time data analysis.
- SQL Statement Executions - Provides capabilities to register tables and execute SQL statements for interfacing with external systems.
- Streaming SQL Transformations - Executes SQL queries directly against live data streams for real-time filtering and aggregation.
- Stream Processing Scaffolds - Provides development scaffolds and connectors to create stream processing applications compatible with multiple engine versions.
- Job Resource Uploaders - Handles the uploading of binaries and configuration dependencies to distributed filesystems prior to job submission.
- Low-Code Automation Platforms - Provides a low-code interface to coordinate the compilation, publishing, and deployment of processing tasks.
- Task Release Tracking - Tracks custom files and task releases to facilitate coordinated updates and rollbacks across multiple versions.
- Task Lifecycle Management - Governs the execution flow of tasks through explicit stages from initialization to destruction.
- Metadata-Driven Orchestration - Uses an external database for metadata-driven orchestration of distributed processing job states and configurations.
- Real-Time Application Performance Monitors - Provides live analysis of application performance metrics and health with second-level monitoring.
- Stream Performance Monitoring - Tracks latency, throughput, and health of distributed data processing pipelines with automated alerts.
