apache/flink-cdc

Features

• Change Data Capture - Captures real-time database modifications and streams them as events to synchronize state with external systems.
• Distributed Stream Processors - Implements a distributed streaming ETL framework for filtering, transforming, and routing data in flight.
• Streaming ETL Pipelines - Provides tools for filtering, transforming, and enriching data in flight as it moves between a source database and a sink.
• Data Integration Pipelines - Provides a pipeline system for orchestrating the movement and routing of data streams between database sources and target sinks.
• Exactly-Once Processing Semantics - Ensures that historical data and change events are processed exactly once, even during job failures.
• Schema Evolution - Detects structural modifications in source tables and automatically applies those changes to the target system.
• Database Synchronization Tools - Synchronizes database changes in real time with automated schema evolution and exactly-once delivery guarantees.
• Distributed Data Synchronization Systems - Moves data from source databases to target systems in real-time or batch mode using a distributed engine.
• Snapshot-to-Log Transitions - Reads an initial database snapshot and transitions to change logs to ensure consistency after failures.
• Real-Time Data Streaming - Allows the construction of custom streaming applications that process and deliver data in real-time.
• Real-time Data Synchronization - Moves data from source databases to target systems in real time to keep downstream environments updated.
• Automated Schema Propagation - Detects structural changes in source databases and automatically applies modifications to downstream target schemas.
• Snapshot Synchronization - Captures historical data using snapshots and transitions to real-time capture to bootstrap synchronization.
• Apache Flink Connectors - A streaming data integration framework that leverages Apache Flink connectors to synchronize database changes.
• Exactly-Once Processing Guarantees - Guarantees that data is written to the target system exactly once to prevent duplicate records.
• Incremental Snapshotting - Reads historical database state while simultaneously capturing real-time change logs without locking tables.
• Full Instance Synchronization - Synchronizes all tables from a source database instance to downstream systems within a single job.
• Cross-Database Data Migrations - Moves entire database instances to data lakes or analytical warehouses using snapshots and change logs.
• Computed Columns - Generates new data columns based on existing fields or metadata using evaluation expressions.
• In-Flight Column Projection - Transforms data in flight by applying evaluation expressions to filter records and generate computed columns.
• Custom Connector Development - Provides interfaces for creating custom source and sink adapters to integrate external systems into data pipelines.
• Multi-Sink Routing - Maps specific source tables to designated sink tables to organize data distribution across multiple target systems.
• Database Layout Extraction - Retrieves namespaces, schemas, and table structures from external systems to identify the current database layout.
• Data Transformation Functions - Removes unnecessary records and modifies data columns using arithmetic, string, and logical functions during synchronization.
• Sink Data Loading - Loads processed data into sink targets such as search engines, data lakes, and analytical databases.
• Source-to-Sink Table Mappings - Defines rules to match source tables to destination tables using one-to-one or pattern-based renaming.
• SQL-Based CDC Integrations - Defines change data capture sources using SQL statements to query and process database changes.
• Table Update Monitoring - Tracks changes to specific database tables to trigger synchronization events.
• Streaming Connector Abstractions - Decouples source and sink implementations from the engine using standardized streaming connector interfaces.
• User-Defined Functions - Integrates custom logic classes to perform specialized data transformations via programmable evaluation methods.
• Event Deserialization - Converts database change events into JSON format with optional schema metadata to optimize processing performance.
• Stream-to-Sink Routing - Routes multi-table data streams to designated sinks using pattern-based renaming and routing rules.
• Pipeline Orchestration - Generates distributed execution operators by translating declarative YAML configuration files into operational streaming jobs.
• Declarative Pipeline Definitions - Defines sources, sinks, and routing rules using structured configuration languages like YAML to deploy jobs.
• Two-Phase Commit Protocols - Ensures exactly-once delivery by coordinating transaction commits between the streaming engine and destination systems.

Star history