apache/flink

Features

• Unified Batch and Stream Processing Engines - Provides a unified runtime that executes both unbounded streaming and bounded batch workloads with consistent semantics.
• Stream Processing - Provides a distributed framework for the continuous ingestion, transformation, and analysis of high-velocity data streams.
• Data Pipelines and ETL - Enables the creation of scalable batch and streaming workloads for analytics and ETL processes.
• Streaming SQL - Provides a SQL interface for executing relational queries and table-based transformations on live data streams.
• Complex Event Processing Engines - Detects temporal patterns and sequences within data streams to trigger real-time actions.
• Data Ingestion Pipelines - Automates the extraction, transformation, and loading of data between brokers, databases, and cloud storage.
• Exactly-Once Processing Semantics - Guarantees that every event is processed exactly once even during system failures through built-in fault tolerance.
• Stream Processing Engines - Executes high-throughput, low-latency computations on real-time data streams with built-in fault tolerance.
• Stateful - Manages persistent operator state to ensure exactly-once processing and consistency during failures.
• Operator State Management - Manages low-level state and time building blocks to implement custom logic for streaming applications.
• Data Serialization Formats - Encodes and decodes data across multiple formats including JSON, CSV, Avro, Parquet, and Protobuf.
• Query Optimizers - Parses SQL and applies optimization rules to generate efficient execution code for faster data retrieval.
• Distributed SQL Querying - Executes relational queries and table-based transformations on live data streams using a distributed SQL engine.
• Event-Time Processing - Groups data using timestamps embedded in records to accurately process out-of-order events.
• External Data Connectors - Provides pluggable connectors to read from and write to message brokers, databases, and cloud storage.
• Local State Stores - Supports pluggable state backends for storing operator state in local memory or remote key-value stores.
• Event-Time Processing - Processes data based on event time rather than arrival time to ensure accuracy with out-of-order streams.
• Persistent State Management - The system stores operator state in pluggable backends to ensure fault tolerance and state recovery.
• Query Processing - Processes data streams and batches using a language-integrated API for selections, filters, and joins.
• Real-Time Analytics - Performs low-latency processing and querying of streaming datasets to provide immediate operational insights.
• Stream Analytics Processing - Groups streaming data into time, count, or session windows to calculate rolling aggregates and metrics.
• Event Pattern Detection - Implements declarative rules to detect temporal patterns and sequences of events within data streams.
• Distributed Consistency Snapshots - Implements Chandy-Lamport snapshotting to ensure exactly-once processing guarantees and consistent recovery from failures.
• Directed Acyclic Graph Engines - Represents computations as a distributed directed acyclic graph of operators executed across a cluster.
• Relational-to-Graph Compilers - Translates relational queries into optimized physical execution graphs of streaming or batch operators.
• Back-pressure Handling - Throttles data ingestion automatically when downstream operators cannot keep up with the processing rate.
• Cluster Query Interfaces - Enables writing and submitting processing queries directly to a cluster via a command-line interface.
• Advanced Analytics Functions - Provides built-in operations for complex data transformations, including graph processing and machine learning libraries.
• Relational Model Extensions - Allows extending the relational model through custom catalogs, formats, connectors, and user-defined functions.
• Python Scripting Environments - Provides a Python interface and runtime to define and execute distributed data processing pipelines.
• Managed Cluster Orchestration - Orchestrates the deployment and maintenance of distributed clusters across managed environments to execute jobs.
• Object Storage Integration - Integrates cloud object storage for scalable and stateless data persistence and as a sink/source.
• Adaptive - Automatically switches between in-memory and out-of-core processing to handle datasets that exceed available physical memory.
• Cluster Resource Managers - Orchestrates the allocation of compute slots and memory across various cluster managers for workload execution.
• Task Progress Monitors - Tracks live execution progress and accumulators through a web-based interface to provide real-time status.
• Job Monitoring Tools - Retrieves the current state and metrics of individual jobs via a REST API.
• Pipeline Execution Visualizers - Displays a graphical representation of the data processing pipeline as it executes to help analyze job flow.
• System Metrics - Reports performance data and system-level telemetry to external monitoring systems to track health and throughput.
• End-to-End Testing - Executes comprehensive tests across the entire architecture to verify system-wide functional correctness before deployment.
• Machine Learning Frameworks - Distributed stream and batch processing framework for data pipelines.
• Big Data - Distributed stream and batch processing engine.
• Big Data Frameworks - Stream and batch processing framework for big data.
• Stream Processing - Processes streams and batches with high performance.
• Data Engineering - Framework for stateful stream and batch processing.
• Data Infrastructure Management - Stream and batch processing framework for large-scale data pipelines.
• Streaming Engines - System for high-throughput, low-latency stateful stream processing.

Star history