# apache/streampark

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/apache-streampark).**

4,312 stars · 1,078 forks · Java · Apache-2.0

## Links

- GitHub: https://github.com/apache/streampark
- Homepage: https://streampark.apache.org/
- awesome-repositories: https://awesome-repositories.com/repository/apache-streampark.md

## Topics

`apache` `development-framework` `easy-to-use` `etl-pipeline` `operation-platform` `streaming` `streampark`

## Description

StreamPark is a centralized management platform designed to coordinate the deployment, monitoring, and operational lifecycle of distributed stream processing and batch applications. It functions as a control plane and orchestrator for data pipelines, specifically providing management capabilities for Apache Flink and Hadoop YARN environments.

The platform distinguishes itself through a low-code approach to task deployment and a multi-engine execution adapter that supports diverse processing runtimes. It facilitates real-time data pipeline management by combining streaming SQL analytics with a resource-based deployment pipeline that handles versioning, binary uploads, and savepoint-based state recovery.

The system covers a broad set of capabilities including distributed job orchestration, real-time data integration via pre-built connectors, and identity integration through LDAP or SSO. It also provides observability tools for second-level application monitoring and automated operational fault notifications.

## Tags

### DevOps & Infrastructure

- [Distributed Job Orchestration](https://awesome-repositories.com/f/devops-infrastructure/distributed-job-orchestration.md) — Provides a centralized framework for linking dependent tasks and managing the full lifecycle of distributed jobs.
- [Centralized Application Maintenance Platforms](https://awesome-repositories.com/f/devops-infrastructure/centralized-application-maintenance-platforms.md) — Coordinates the debugging, deployment, and maintenance of streaming applications through a centralized management platform. ([source](https://cdn.jsdelivr.net/gh/apache/streampark@dev/README.md))
- [Hub-and-Spoke Control Planes](https://awesome-repositories.com/f/devops-infrastructure/distributed-deployment/hub-and-spoke-control-planes.md) — Implements a centralized control plane to manage the deployment and monitoring of batch and stream tasks.
- [Stream Processing Coordinators](https://awesome-repositories.com/f/devops-infrastructure/stream-processing-coordinators.md) — Coordinates the deployment and lifecycle of streaming and batch applications from a single centralized control point. ([source](https://streampark.apache.org/))
- [Centralized Control Planes](https://awesome-repositories.com/f/devops-infrastructure/stream-processing-pipeline-deployments/centralized-control-planes.md) — Provides a centralized control interface for deploying and operating real-time data pipelines across multiple engines.
- [Task Lifecycle Management](https://awesome-repositories.com/f/devops-infrastructure/automation-orchestration/task-execution-frameworks/task-job-management/task-lifecycle-management.md) — Governs the execution flow of tasks through explicit stages from initialization to destruction. ([source](https://streampark.apache.org/docs/framework/dev-model))
- [Metadata-Driven Orchestration](https://awesome-repositories.com/f/devops-infrastructure/job-priority-management/priority-based-job-schedulers/job-execution-controllers/metadata-driven-orchestration.md) — Uses an external database for metadata-driven orchestration of distributed processing job states and configurations.

### Data & Databases

- [Apache Flink Management Platforms](https://awesome-repositories.com/f/data-databases/apache-flink-management-platforms.md) — Provides a centralized control plane for deploying, monitoring, and managing Apache Flink applications.
- [Execution Engine Translation](https://awesome-repositories.com/f/data-databases/data-pipeline-orchestration/data-engineering-pipelines/execution-engine-translation.md) — Translates high-level task definitions into engine-specific configurations via an execution engine translation layer.
- [Unified Batch and Stream Processing Engines](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/data-processing-frameworks/unified-batch-and-stream-processing-engines.md) — Executes both real-time streaming and batch workloads across different versions of processing engines. ([source](https://streampark.apache.org/))
- [Low-Code Pipeline Managers](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/data-processing/distributed-processing-frameworks/real-time-data-processors/low-code-pipeline-managers.md) — Offers a low-code platform for integrating data sources, executing streaming SQL, and managing warehouse connections.
- [Stream and Pipeline Orchestration](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/data-transformation/stream-pipeline-orchestration.md) — Coordinates the flow, transformation, and distributed processing of continuous data streams across multiple engine versions.
- [Distributed Task Schedulers](https://awesome-repositories.com/f/data-databases/distributed-task-schedulers.md) — Orchestrates and distributes complex data processing workflows across computing clusters. ([source](https://streampark.apache.org/docs/framework/quick-start))
- [Real-Time Data Integration Platforms](https://awesome-repositories.com/f/data-databases/real-time-data-integration-platforms.md) — Synchronizes live data across heterogeneous storage and analytical environments using pre-built connectors.
- [Streaming State Recovery](https://awesome-repositories.com/f/data-databases/undo-logs/transaction-recovery-logs/log-state-recovery/streaming-state-recovery.md) — Recovers incremental operator states and output results from durable storage after failures. ([source](https://streampark.apache.org/docs/framework/quick-start))
- [Development Scaffolding](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/stream-processing-systems/stream-processing/development-scaffolding.md) — Provides standardized configurations and development scaffolds to implement streaming logic across multiple processing engines. ([source](https://streampark.apache.org/docs/get-started/introduction))
- [Data Processing Tasks](https://awesome-repositories.com/f/data-databases/data-processing-tasks.md) — Handles compute-intensive operations including startup, savepoints, and performance analysis of stream jobs. ([source](https://streampark.apache.org/docs/get-started/introduction))
- [Data Warehouse Integrations](https://awesome-repositories.com/f/data-databases/data-warehouse-integrations.md) — Links processing engines with open-source data lakes and real-time warehouses to streamline storage and retrieval. ([source](https://streampark.apache.org/))
- [Hadoop Workflow Orchestrators](https://awesome-repositories.com/f/data-databases/hadoop-workflow-orchestrators.md) — Schedules and monitors big data workloads running on Hadoop clusters via HDFS.
- [Real-Time Data Streaming](https://awesome-repositories.com/f/data-databases/real-time-data-streaming.md) — Executes interactive streaming queries and manages streaming warehouses for real-time data analysis. ([source](https://cdn.jsdelivr.net/gh/apache/streampark@dev/README.md))
- [SQL Statement Executions](https://awesome-repositories.com/f/data-databases/sql-statement-parsing/sql-statement-executions.md) — Provides capabilities to register tables and execute SQL statements for interfacing with external systems. ([source](https://streampark.apache.org/docs/framework/dev-model))
- [Streaming SQL Transformations](https://awesome-repositories.com/f/data-databases/streaming-sql-transformations.md) — Executes SQL queries directly against live data streams for real-time filtering and aggregation.

### Development Tools & Productivity

- [Unified Web Interfaces](https://awesome-repositories.com/f/development-tools-productivity/application-lifecycle-management/unified-web-interfaces.md) — Provides a unified web interface for building, deploying, and operating streaming applications across various engines. ([source](https://streampark.apache.org/docs/get-started/intro))
- [Stream Processing Scaffolds](https://awesome-repositories.com/f/development-tools-productivity/application-scaffolding-tools/stream-processing-scaffolds.md) — Provides development scaffolds and connectors to create stream processing applications compatible with multiple engine versions. ([source](https://cdn.jsdelivr.net/gh/apache/streampark@dev/README.md))
- [Job Resource Uploaders](https://awesome-repositories.com/f/development-tools-productivity/job-scheduling-resources/job-resource-uploaders.md) — Handles the uploading of binaries and configuration dependencies to distributed filesystems prior to job submission.
- [Low-Code Automation Platforms](https://awesome-repositories.com/f/development-tools-productivity/low-code-automation-platforms.md) — Provides a low-code interface to coordinate the compilation, publishing, and deployment of processing tasks. ([source](https://streampark.apache.org/docs/get-started/introduction))
- [Task Release Tracking](https://awesome-repositories.com/f/development-tools-productivity/platform-versioning-tools/pipeline-task-versioning/task-release-tracking.md) — Tracks custom files and task releases to facilitate coordinated updates and rollbacks across multiple versions. ([source](https://streampark.apache.org/docs/framework/quick-start))

### System Administration & Monitoring

- [Application Lifecycle Management](https://awesome-repositories.com/f/system-administration-monitoring/application-lifecycle-management.md) — Manages the operational state, version updates, and health of running distributed stream processing tasks.
- [Real-Time Application Performance Monitors](https://awesome-repositories.com/f/system-administration-monitoring/application-performance-monitoring/application-performance-monitoring/real-time-application-performance-monitors.md) — Provides live analysis of application performance metrics and health with second-level monitoring. ([source](https://streampark.apache.org/))
- [Stream Performance Monitoring](https://awesome-repositories.com/f/system-administration-monitoring/stream-performance-monitoring.md) — Tracks latency, throughput, and health of distributed data processing pipelines with automated alerts.

### Part of an Awesome List

- [YARN and Cluster Management](https://awesome-repositories.com/f/awesome-lists/devops/yarn-and-cluster-management.md) — Manages resources and executes streaming jobs within Hadoop YARN clusters. ([source](https://streampark.apache.org/docs/get-started/installation))
- [Data Connectors](https://awesome-repositories.com/f/awesome-lists/devtools/data-connectors.md) — Utilizes pre-built connectors to efficiently acquire and move data between various sources and destinations. ([source](https://streampark.apache.org/docs/get-started/introduction))