Awesome GitHub RepositoriesChange Data Capture Tools

This project is an enterprise-grade Java framework designed for building scalable, full-stack e-commerce applications. It provides a comprehensive foundation for microservice-based distributed architectures, enabling the development of complex retail platforms that include product management, order processing, and secure user authentication. By leveraging modular service patterns and centralized API gateways, the framework supports the construction of resilient systems that decompose monolithic business logic into independent, manageable services. The platform distinguishes itself through a r

TiDB is a horizontally scalable, distributed SQL database designed to provide consistent transactional storage and high-performance analytical processing within a single unified architecture. It utilizes a decoupled compute-storage design and a distributed key-value storage layer to ensure horizontal scalability and efficient range-based queries. By employing a consensus-based replication algorithm, the system maintains high availability and automatic failover across multiple nodes and geographical regions. The platform distinguishes itself through its hybrid transactional and analytical proc

Canal is a database replication middleware that performs change data capture by simulating a database replica. It monitors transaction logs to stream incremental data modifications to downstream systems in real time, acting as an event streaming infrastructure that transforms low-level binary logs into structured, consumable message streams. The project distinguishes itself through a high-throughput architecture that utilizes concurrent multi-threaded parsing and stateful log position tracking to ensure reliable data delivery. It employs a pluggable sink architecture that decouples data extra

Airbyte is a data integration platform designed to synchronize information between diverse applications, databases, and data warehouses. It functions as an extract, transform, and load orchestrator that manages automated data movement workflows across cloud, on-premise, and hybrid environments. The platform provides a standardized interface for connectors, enabling the movement of structured and unstructured data while maintaining stateful checkpoints for reliable incremental syncing. The platform distinguishes itself through a containerized architecture that isolates connectors to prevent de

SeaTunnel is a distributed data integration engine designed to synchronize structured and unstructured data across diverse sources and sinks. It functions as a multi-engine execution framework that can run data integration tasks across different distributed computing backends to optimize workload performance. The project is distinguished by a visual data pipeline designer for configuring workflows without manual code and a specialized change data capture tool for streaming incremental database updates. It also includes an enrichment pipeline that integrates large language models and embedding

Benthos is a stream processing engine and data integration pipeline used for routing, transforming, and connecting data streams between diverse sources and sinks. It functions as event routing middleware and a change data capture tool, streaming real-time database modifications as discrete events for downstream processing. The system utilizes a declarative pipeline configuration, where data flow and processing logic are defined in a single static file. It features a specialized domain-specific language for mapping, filtering, and enriching data payloads, allowing for complex transformations w

Connect is a Kafka data integration platform and stream processing engine used to build declarative pipelines that move and transform messages between Kafka topics and external sources. It functions as a Kafka Connect framework and a change data capture tool, streaming real-time database modifications to synchronize data across distributed environments. The project differentiates itself through a dedicated mapping language for mutating and reshaping message payloads and the ability to execute custom processing logic within a sandboxed WebAssembly runtime. It also provides an observability pip

Otter is a distributed database synchronization system and change data capture tool designed to replicate data between databases across multiple geographic regions. It functions as a synchronization orchestrator and ETL data pipeline that mirrors records and associated files in real time. The system employs incremental log parsing to capture database changes and utilizes a consistency-based convergence algorithm and loop-avoidance logic to manage bi-directional replication. It processes data through a pipeline of selection, extraction, transformation, and loading to handle joins and format co

Maxwell is a MySQL change data capture tool and binlog streaming application that converts database modifications into structured JSON events. It functions as a data pipeline that reads MySQL binary logs to synchronize changes across external indices, search engines, and distributed messaging systems such as Kafka. The project provides capabilities to maintain persistent audit trails by recording a chronological history of all database modifications. It enables real-time data synchronization and event-driven architecture integration by streaming database changes to external platforms to trigg

Chunjun ist ein verteiltes Datenintegrations-Framework und eine SQL-basierte ETL-Pipeline, die entwickelt wurde, um Daten zwischen heterogenen Quellen zu synchronisieren. Es fungiert als Change-Data-Capture-Tool (CDC) und heterogener Datensynchronisierer und nutzt eine verteilte Verarbeitungsumgebung, um Daten über verschiedene Datenbanktypen hinweg zu bewegen und zu transformieren. Das System zeichnet sich durch seine Plugin-basierte Connector-Architektur aus, die die Entwicklung benutzerdefinierter Source- und Sink-Plugins ermöglicht, um die Konnektivität auf nicht unterstützte Datensysteme auszuweiten. Es unterstützt Echtzeit-Change-Data-Capture aus relationalen Datenbank-Logs und implementiert Schema-Evolution-Propagation, um strukturelle Änderungen automatisch von Quell- auf Zieltabellen anzuwenden. Das Framework bietet Funktionen für inkrementelle Datensynchronisierung und quellübergreifende Datenberechnung mittels SQL-Logik. Die Zuverlässigkeit wird durch Checkpoint-basiertes Task-Recovery zur Wiederaufnahme unterbrochener Übertragungen und Dead-Letter-Queues für das Management fehlerhafter Daten zur Prüfung falsch formatierter Datensätze verwaltet. Integrationsaufgaben können über eigenständige Cluster, Yarn oder Kubernetes-Umgebungen bereitgestellt werden, mit Unterstützung für containerisierte Bereitstellung via Docker.