# zhisheng17/flink-learning

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/zhisheng17-flink-learning).**

15,071 stars · 3,943 forks · Java · Apache-2.0

## Links

- GitHub: https://github.com/zhisheng17/flink-learning
- Homepage: http://www.54tianzhisheng.cn/tags/Flink/
- awesome-repositories: https://awesome-repositories.com/repository/zhisheng17-flink-learning.md

## Topics

`clickhouse` `elasticsearch` `flink` `hbase` `influxdb` `kafka` `loki` `mysql` `opentsdb` `rabbitmq` `redis` `rocketmq` `spark` `stream-processing` `streaming`

## Description

This project is a collection of educational resources and reference implementations for the Apache Flink stream processing framework. It provides a learning resource focused on mastering distributed stream processing through implementation guides, performance tuning tutorials, and practical examples.

The repository features detailed walkthroughs for building real-time data pipelines using the DataStream and Table APIs. It includes specific integration examples for connecting Apache Flink with Kafka brokers and Elasticsearch indices, as well as reference implementations for real-time deduplication and fault-tolerant state management.

The project covers a broad range of stream processing capabilities, including windowed aggregations, complex data transformations, and declarative SQL execution. It also provides guidance on cluster management, high availability configuration, and operational monitoring via the web interface.

The content is presented as a series of guides and examples to assist with optimizing resource allocation, parallelism, and pipeline throughput.

## Tags

### Education & Learning Resources

- [Stream Processing Implementation Guides](https://awesome-repositories.com/f/education-learning-resources/stream-processing-implementation-guides.md) — Provides detailed walkthroughs for building real-time data pipelines using the DataStream and Table APIs.
- [Stream Processing Learning Resources](https://awesome-repositories.com/f/education-learning-resources/stream-processing-learning-resources.md) — Offers a collection of tutorials and practical examples for mastering the Apache Flink framework.
- [Stream Processing Performance Tuning Guides](https://awesome-repositories.com/f/education-learning-resources/stream-processing-performance-tuning-guides.md) — Provides guides on optimizing resource allocation and parallelism to increase throughput and reduce latency.

### Data & Databases

- [Windowed Event Aggregations](https://awesome-repositories.com/f/data-databases/data-processing-pipelines/data-transformation/data-aggregation-tools/github-api-aggregators/windowed-event-aggregations.md) — Groups continuous data flows into temporal or count-based windows to perform periodic aggregations.
- [Data Stream Integrations](https://awesome-repositories.com/f/data-databases/data-stream-integrations.md) — Enables the integration of data flow between external storage systems and messaging queues across environments. ([source](http://www.54tianzhisheng.cn/2018/09/18/flink-install))
- [Data Transformation](https://awesome-repositories.com/f/data-databases/data-transformation.md) — Implements complex data transformation using mapping, filtering, and joining operations on data streams. ([source](http://www.54tianzhisheng.cn/2018/10/13/flink-introduction/))
- [Stream Processing Pipelines](https://awesome-repositories.com/f/data-databases/data-transformation-pipelines/stream-processing-pipelines.md) — Provides reference implementations for constructing end-to-end real-time data pipelines. ([source](http://www.54tianzhisheng.cn/2018/10/13/flink-introduction/))
- [Distributed Execution Coordinators](https://awesome-repositories.com/f/data-databases/distributed-execution-coordinators.md) — Manages task scheduling and failure recovery across a distributed cluster of job and task managers. ([source](http://www.54tianzhisheng.cn/2018/10/13/flink-introduction/))
- [Distributed Task Schedulers](https://awesome-repositories.com/f/data-databases/distributed-task-schedulers.md) — Orchestrates the allocation of parallel operator instances across a distributed cluster of job and task managers.
- [Event-Time Processing](https://awesome-repositories.com/f/data-databases/event-time-processing.md) — Guides the use of event-time processing to group data using system, ingestion, or event timestamps. ([source](http://www.54tianzhisheng.cn/2018/12/11/Flink-time/))
- [Time-Window Aggregations](https://awesome-repositories.com/f/data-databases/event-time-processing/time-window-aggregations.md) — Implements time-window aggregations to perform computations on data grouped into temporal intervals. ([source](http://www.54tianzhisheng.cn/2018/12/08/Flink-Stream-Windows/))
- [External Data Connectors](https://awesome-repositories.com/f/data-databases/external-data-connectors.md) — Provides frameworks for integrating and hosting external data streams through specialized source and sink interfaces.
- [External Data Ingestion](https://awesome-repositories.com/f/data-databases/external-data-ingestion.md) — Reads data from external systems to create a continuous stream for real-time processing. ([source](http://www.54tianzhisheng.cn/2018/10/30/flink-create-source/))
- [Key-Based Partitioning](https://awesome-repositories.com/f/data-databases/indexing-and-search/index-based-pruning/key-based-partitioning.md) — Implements key-based partitioning to ensure records with the same key reach the same operator. ([source](http://www.54tianzhisheng.cn/2018/11/04/Flink-Data-transformation/))
- [Reference Implementations](https://awesome-repositories.com/f/data-databases/real-time-analytics/stream-analytics-processing/reference-implementations.md) — Implements reference examples for real-time deduplication, windowed aggregations, and fault-tolerant state management.
- [Real-Time Data Aggregators](https://awesome-repositories.com/f/data-databases/real-time-data-aggregators.md) — Provides real-time data aggregation capabilities for computing sums and reductions across keyed partitions. ([source](http://www.54tianzhisheng.cn/2018/11/04/Flink-Data-transformation/))
- [Real-Time Data Streaming](https://awesome-repositories.com/f/data-databases/real-time-data-streaming.md) — Facilitates real-time data streaming and analysis using time windows to manage out-of-order events. ([source](https://cdn.jsdelivr.net/gh/zhisheng17/flink-learning@master/README.md))
- [SQL Query Execution](https://awesome-repositories.com/f/data-databases/sql-query-execution.md) — Translates high-level SQL table and query definitions into an optimized graph of streaming operators for execution.
- [State Checkpointing](https://awesome-repositories.com/f/data-databases/state-checkpointing.md) — Implements state backends and periodic checkpointing to ensure consistent recovery of streaming applications after failures. ([source](https://cdn.jsdelivr.net/gh/zhisheng17/flink-learning@master/README.md))
- [Stream Transformations](https://awesome-repositories.com/f/data-databases/stream-transformations.md) — Demonstrates stream transformations using mapping and flattening functions to modify record data. ([source](http://www.54tianzhisheng.cn/2018/11/04/Flink-Data-transformation/))
- [Streaming Source and Sink Integration](https://awesome-repositories.com/f/data-databases/streaming-source-and-sink-integration.md) — Implements connectivity between processing jobs and external systems for data ingestion and egress. ([source](https://cdn.jsdelivr.net/gh/zhisheng17/flink-learning@master/README.md))
- [Streaming State Recovery](https://awesome-repositories.com/f/data-databases/undo-logs/transaction-recovery-logs/log-state-recovery/streaming-state-recovery.md) — Implements recovery of incremental operator states and output results from durable remote storage after failures.
- [Watermark-Based Event Tracking](https://awesome-repositories.com/f/data-databases/watermark-based-event-tracking.md) — Tracks event-time progress using watermarks to handle out-of-order data and trigger window computations.
- [Count-Based Windowing](https://awesome-repositories.com/f/data-databases/count-based-windowing.md) — Provides implementations for performing computations over count-based intervals in data streams. ([source](http://www.54tianzhisheng.cn/2018/12/08/Flink-Stream-Windows/))
- [Data Destination Connectors](https://awesome-repositories.com/f/data-databases/data-processing-configurations/data-destination-connectors.md) — Outputs processed data streams to external systems such as message queues, databases, or files. ([source](http://www.54tianzhisheng.cn/2018/10/29/flink-sink/))
- [Data Sinking](https://awesome-repositories.com/f/data-databases/data-sinking.md) — Provides interfaces to define how records are written to non-standard external storage systems. ([source](http://www.54tianzhisheng.cn/2018/10/29/flink-sink/))
- [Elasticsearch Exporters](https://awesome-repositories.com/f/data-databases/database-connectivity/elasticsearch-connectors/elasticsearch-exporters.md) — Streams processed document data into Elasticsearch indices using configurable batch sizes and host resolution. ([source](http://www.54tianzhisheng.cn/2018/12/30/Flink-ElasticSearch-Sink/))
- [MySQL Sinks](https://awesome-repositories.com/f/data-databases/database-connectivity/mysql-connectors/mysql-sinks.md) — Persists processed records into MySQL tables using connection pooling and batch execution. ([source](http://www.54tianzhisheng.cn/2019/01/15/Flink-MySQL-sink/))
- [Custom Windowing Logic](https://awesome-repositories.com/f/data-databases/event-time-processing/time-window-aggregations/custom-windowing-logic.md) — Demonstrates how to implement custom windowing logic for flexible stream aggregations. ([source](http://www.54tianzhisheng.cn/2018/09/18/flink-install))
- [File-Based Data Import](https://awesome-repositories.com/f/data-databases/file-based-data-import.md) — Reads text or formatted files from a path to process data once or continuously as a stream. ([source](http://www.54tianzhisheng.cn/2018/10/28/flink-sources/))
- [High Availability Configurations](https://awesome-repositories.com/f/data-databases/high-availability-configurations.md) — Configures metadata persistence to ensure automatic system recovery from manager failures. ([source](http://www.54tianzhisheng.cn/2018/10/27/flink-config/))
- [Kafka and Elasticsearch Integrations](https://awesome-repositories.com/f/data-databases/kafka-and-elasticsearch-integrations.md) — Provides practical code examples for connecting Apache Flink with Kafka brokers and Elasticsearch indices.
- [Topic Consumption](https://awesome-repositories.com/f/data-databases/kafka-message-browsers/kafka-schema-deserializers/topic-consumption.md) — Reads streaming data from Kafka topics and converts raw strings into structured objects for application use. ([source](http://www.54tianzhisheng.cn/2019/01/15/Flink-MySQL-sink/))
- [Kafka Stream Exporters](https://awesome-repositories.com/f/data-databases/real-time-data-streaming/kafka-stream-exporters.md) — Sends processed data streams to Kafka message topics using specific broker lists and serialization schemas. ([source](http://www.54tianzhisheng.cn/2019/01/06/Flink-Kafka-sink/))
- [Resource Allocation](https://awesome-repositories.com/f/data-databases/resource-allocation.md) — Configures hardware and memory requirements, including task slots, to distribute workloads across compute nodes. ([source](http://www.54tianzhisheng.cn/2018/10/27/flink-config/))
- [Window Triggering and Eviction](https://awesome-repositories.com/f/data-databases/window-triggering-and-eviction.md) — Explains how to specify element assignment, trigger conditions, and eviction policies for stream windows. ([source](http://www.54tianzhisheng.cn/2018/12/08/Flink-Stream-Windows/))

### Software Engineering & Architecture

- [Cluster Load Balancing](https://awesome-repositories.com/f/software-engineering-architecture/cluster-load-balancing.md) — Adjusts parallelism and slot allocation to distribute processing tasks evenly across connected servers. ([source](http://www.54tianzhisheng.cn/2018/09/18/flink-install))
- [Recoverable State Management](https://awesome-repositories.com/f/software-engineering-architecture/fault-tolerance/recoverable-state-management.md) — Implements fault-tolerant state management within process functions to enable complex computations across time. ([source](http://www.54tianzhisheng.cn/2018/10/13/flink-introduction/))
- [Failure Handling Policies](https://awesome-repositories.com/f/software-engineering-architecture/failure-handling-policies.md) — Implements recovery logic to handle malformed requests and indexing failures without crashing pipelines. ([source](http://www.54tianzhisheng.cn/2018/12/30/Flink-ElasticSearch-Sink/))
- [Retry Policies](https://awesome-repositories.com/f/software-engineering-architecture/retry-policies.md) — Applies backoff strategies and retry limits to prevent data loss during failed bulk write operations. ([source](http://www.54tianzhisheng.cn/2018/12/30/Flink-ElasticSearch-Sink/))
- [Resource Slot Scheduling](https://awesome-repositories.com/f/software-engineering-architecture/task-scheduling/parallel-task-executors/resource-slot-scheduling.md) — Controls parallelism by dividing task manager memory into fixed resource slots to isolate subtasks on a node.

### Web Development

- [Stream Join Operators](https://awesome-repositories.com/f/web-development/asynchronous-api-clients/stream-composition/stream-combinators/stream-join-operators.md) — Implements stream join operators to merge multiple unbounded data streams based on shared keys. ([source](http://www.54tianzhisheng.cn/2018/11/04/Flink-Data-transformation/))

### Artificial Intelligence & ML

- [Custom Data Sources](https://awesome-repositories.com/f/artificial-intelligence-ml/feature-stores/custom-data-sources.md) — Provides source functions and interfaces to fetch data from non-standard backends or database tables. ([source](http://www.54tianzhisheng.cn/2018/10/30/flink-create-source/))

### DevOps & Infrastructure

- [Application Cluster Deployments](https://awesome-repositories.com/f/devops-infrastructure/container-cluster-deployments/application-cluster-deployments.md) — Supports deploying applications across standalone clusters or managed cloud environments. ([source](http://www.54tianzhisheng.cn/2018/10/13/flink-introduction/))
- [Data Throughput Optimizers](https://awesome-repositories.com/f/devops-infrastructure/data-throughput-optimizers.md) — Offers tutorials on using data throughput optimizers to increase data movement and reduce latency. ([source](http://www.54tianzhisheng.cn/2019/11/15/flink-in-action/](http://www.54tianzhisheng.cn/2019/11/15/flink-in-action/))
- [Parallel Execution Settings](https://awesome-repositories.com/f/devops-infrastructure/parallel-execution-settings.md) — Provides guides for configuring parallel execution settings to optimize stream processing speed. ([source](http://www.54tianzhisheng.cn/2019/01/14/Flink-parallelism-slot/))

### System Administration & Monitoring

- [Task Execution Monitoring](https://awesome-repositories.com/f/system-administration-monitoring/activity-monitors/task-execution-monitoring.md) — Tracks task metrics and reviews system logs via a dashboard to verify processing flow. ([source](http://www.54tianzhisheng.cn/2019/01/05/Flink-run/))
- [Web Service Monitoring](https://awesome-repositories.com/f/system-administration-monitoring/administrative-operations/service-and-infrastructure-management/service-infrastructure-monitoring/service-monitors/web-service-monitoring.md) — Exposes a web-based runtime monitor to observe and manage active processing tasks. ([source](http://www.54tianzhisheng.cn/2018/10/27/flink-config/))
- [Application Metric Tracking](https://awesome-repositories.com/f/system-administration-monitoring/application-metric-tracking.md) — Tracks operational performance and system metrics using integrations with external monitoring tools. ([source](http://www.54tianzhisheng.cn/2018/09/18/flink-install))