apache/flink-cdc
Flink Cdc
This project is a streaming data integration framework that captures real-time database changes and synchronizes them with downstream systems. It operates as a distributed streaming ETL and database synchronizer, reading database logs and snapshots to propagate row-level modifications to target sinks.
The system supports declarative data integration, allowing users to define source-to-sink data flows using SQL or YAML configurations. It distinguishes itself by automating schema evolution to maintain synchronization when source structures change and ensuring exactly-once delivery and processing guarantees to prevent duplicate records.
Broad capabilities include distributed data synchronization, multi-sink routing, and in-flight data transformation. The framework provides tools for filtering records, generating computed columns, and performing non-blocking incremental snapshotting to capture historical state without locking tables.
Applications can be packaged into deployable archives containing the necessary connectors for distributed execution.
Features
- Change Data Capture - Captures real-time database modifications and streams them as events to synchronize state with external systems.
- Distributed Stream Processors - Implements a distributed streaming ETL framework for filtering, transforming, and routing data in flight.
- Streaming ETL Pipelines - Provides tools for filtering, transforming, and enriching data in flight as it moves between a source database and a sink.
- Data Integration Pipelines - Provides a pipeline system for orchestrating the movement and routing of data streams between database sources and target sinks.
- Exactly-Once Processing Semantics - Ensures that historical data and change events are processed exactly once, even during job failures.
- Schema Evolution - Detects structural modifications in source tables and automatically applies those changes to the target system.
- Database Synchronization Tools - Synchronizes database changes in real time with automated schema evolution and exactly-once delivery guarantees.
- Distributed Data Synchronization Systems - Moves data from source databases to target systems in real-time or batch mode using a distributed engine.
- Snapshot-to-Log Transitions - Reads an initial database snapshot and transitions to change logs to ensure consistency after failures.
- Real-Time Data Streaming - Allows the construction of custom streaming applications that process and deliver data in real-time.
- Real-time Data Synchronization - Moves data from source databases to target systems in real time to keep downstream environments updated.
- Automated Schema Propagation - Detects structural changes in source databases and automatically applies modifications to downstream target schemas.
- Snapshot Synchronization - Captures historical data using snapshots and transitions to real-time capture to bootstrap synchronization.
- Apache Flink Connectors - A streaming data integration framework that leverages Apache Flink connectors to synchronize database changes.
- Exactly-Once Processing Guarantees - Guarantees that data is written to the target system exactly once to prevent duplicate records.
- Incremental Snapshotting - Reads historical database state while simultaneously capturing real-time change logs without locking tables.
- Full Instance Synchronization - Synchronizes all tables from a source database instance to downstream systems within a single job.
- Cross-Database Data Migrations - Moves entire database instances to data lakes or analytical warehouses using snapshots and change logs.
- Computed Columns - Generates new data columns based on existing fields or metadata using evaluation expressions.
- In-Flight Column Projection - Transforms data in flight by applying evaluation expressions to filter records and generate computed columns.
- Custom Connector Development - Provides interfaces for creating custom source and sink adapters to integrate external systems into data pipelines.
- Multi-Sink Routing - Maps specific source tables to designated sink tables to organize data distribution across multiple target systems.
- Database Layout Extraction - Retrieves namespaces, schemas, and table structures from external systems to identify the current database layout.
- Data Transformation Functions - Removes unnecessary records and modifies data columns using arithmetic, string, and logical functions during synchronization.
- Sink Data Loading - Loads processed data into sink targets such as search engines, data lakes, and analytical databases.
- Source-to-Sink Table Mappings - Defines rules to match source tables to destination tables using one-to-one or pattern-based renaming.
- SQL-Based CDC Integrations - Defines change data capture sources using SQL statements to query and process database changes.
- Table Update Monitoring - Tracks changes to specific database tables to trigger synchronization events.
- Streaming Connector Abstractions - Decouples source and sink implementations from the engine using standardized streaming connector interfaces.
- User-Defined Functions - Integrates custom logic classes to perform specialized data transformations via programmable evaluation methods.
- Event Deserialization - Converts database change events into JSON format with optional schema metadata to optimize processing performance.
- Stream-to-Sink Routing - Routes multi-table data streams to designated sinks using pattern-based renaming and routing rules.
- Pipeline Orchestration - Generates distributed execution operators by translating declarative YAML configuration files into operational streaming jobs.
- Declarative Pipeline Definitions - Defines sources, sinks, and routing rules using structured configuration languages like YAML to deploy jobs.
- Two-Phase Commit Protocols - Ensures exactly-once delivery by coordinating transaction commits between the streaming engine and destination systems.
