apache/streampark
Streampark
StreamPark es una plataforma de gestión centralizada diseñada para coordinar el despliegue, monitoreo y ciclo de vida operativo de aplicaciones de procesamiento de flujos distribuidos y procesamiento por lotes (batch). Funciona como un plano de control y orquestador para pipelines de datos, proporcionando específicamente capacidades de gestión para entornos Apache Flink y Hadoop YARN.
La plataforma se distingue por un enfoque de bajo código para el despliegue de tareas y un adaptador de ejecución multi-motor que admite diversos runtimes de procesamiento. Facilita la gestión de pipelines de datos en tiempo real combinando análisis SQL de streaming con un pipeline de despliegue basado en recursos que maneja el versionado, subidas de binarios y recuperación de estado basada en savepoints.
El sistema cubre un amplio conjunto de capacidades, incluyendo orquestación de trabajos distribuidos, integración de datos en tiempo real a través de conectores preconstruidos e integración de identidad a través de LDAP o SSO. También proporciona herramientas de observabilidad para el monitoreo de aplicaciones de segundo nivel y notificaciones operativas automatizadas de fallos.
Features
- Distributed Job Orchestration - Provides a centralized framework for linking dependent tasks and managing the full lifecycle of distributed jobs.
- Apache Flink Management Platforms - Provides a centralized control plane for deploying, monitoring, and managing Apache Flink applications.
- Execution Engine Translation - Translates high-level task definitions into engine-specific configurations via an execution engine translation layer.
- Unified Batch and Stream Processing Engines - Executes both real-time streaming and batch workloads across different versions of processing engines.
- Low-Code Pipeline Managers - Offers a low-code platform for integrating data sources, executing streaming SQL, and managing warehouse connections.
- Stream and Pipeline Orchestration - Coordinates the flow, transformation, and distributed processing of continuous data streams across multiple engine versions.
- Distributed Task Schedulers - Orchestrates and distributes complex data processing workflows across computing clusters.
- Real-Time Data Integration Platforms - Synchronizes live data across heterogeneous storage and analytical environments using pre-built connectors.
- Streaming State Recovery - Recovers incremental operator states and output results from durable storage after failures.
- Unified Web Interfaces - Provides a unified web interface for building, deploying, and operating streaming applications across various engines.
- Centralized Application Maintenance Platforms - Coordinates the debugging, deployment, and maintenance of streaming applications through a centralized management platform.
- Hub-and-Spoke Control Planes - Implements a centralized control plane to manage the deployment and monitoring of batch and stream tasks.
- Stream Processing Coordinators - Coordinates the deployment and lifecycle of streaming and batch applications from a single centralized control point.
- Centralized Control Planes - Provides a centralized control interface for deploying and operating real-time data pipelines across multiple engines.
- Application Lifecycle Management - Manages the operational state, version updates, and health of running distributed stream processing tasks.
- YARN and Cluster Management - Manages resources and executes streaming jobs within Hadoop YARN clusters.
- Data Connectors - Utilizes pre-built connectors to efficiently acquire and move data between various sources and destinations.
- Development Scaffolding - Provides standardized configurations and development scaffolds to implement streaming logic across multiple processing engines.
- Data Processing Tasks - Handles compute-intensive operations including startup, savepoints, and performance analysis of stream jobs.
- Data Warehouse Integrations - Links processing engines with open-source data lakes and real-time warehouses to streamline storage and retrieval.
- Hadoop Workflow Orchestrators - Schedules and monitors big data workloads running on Hadoop clusters via HDFS.
- Real-Time Data Streaming - Executes interactive streaming queries and manages streaming warehouses for real-time data analysis.
- SQL Statement Executions - Provides capabilities to register tables and execute SQL statements for interfacing with external systems.
- Streaming SQL Transformations - Executes SQL queries directly against live data streams for real-time filtering and aggregation.
- Stream Processing Scaffolds - Provides development scaffolds and connectors to create stream processing applications compatible with multiple engine versions.
- Job Resource Uploaders - Handles the uploading of binaries and configuration dependencies to distributed filesystems prior to job submission.
- Low-Code Automation Platforms - Provides a low-code interface to coordinate the compilation, publishing, and deployment of processing tasks.
- Task Release Tracking - Tracks custom files and task releases to facilitate coordinated updates and rollbacks across multiple versions.
- Task Lifecycle Management - Governs the execution flow of tasks through explicit stages from initialization to destruction.
- Metadata-Driven Orchestration - Uses an external database for metadata-driven orchestration of distributed processing job states and configurations.
- Real-Time Application Performance Monitors - Provides live analysis of application performance metrics and health with second-level monitoring.
- Stream Performance Monitoring - Tracks latency, throughput, and health of distributed data processing pipelines with automated alerts.
