Awesome GitHub RepositoriesChange Data Capture Tools

This project is an enterprise-grade Java framework designed for building scalable, full-stack e-commerce applications. It provides a comprehensive foundation for microservice-based distributed architectures, enabling the development of complex retail platforms that include product management, order processing, and secure user authentication. By leveraging modular service patterns and centralized API gateways, the framework supports the construction of resilient systems that decompose monolithic business logic into independent, manageable services. The platform distinguishes itself through a r

TiDB is a horizontally scalable, distributed SQL database designed to provide consistent transactional storage and high-performance analytical processing within a single unified architecture. It utilizes a decoupled compute-storage design and a distributed key-value storage layer to ensure horizontal scalability and efficient range-based queries. By employing a consensus-based replication algorithm, the system maintains high availability and automatic failover across multiple nodes and geographical regions. The platform distinguishes itself through its hybrid transactional and analytical proc

Canal is a database replication middleware that performs change data capture by simulating a database replica. It monitors transaction logs to stream incremental data modifications to downstream systems in real time, acting as an event streaming infrastructure that transforms low-level binary logs into structured, consumable message streams. The project distinguishes itself through a high-throughput architecture that utilizes concurrent multi-threaded parsing and stateful log position tracking to ensure reliable data delivery. It employs a pluggable sink architecture that decouples data extra

Airbyte is a data integration platform designed to synchronize information between diverse applications, databases, and data warehouses. It functions as an extract, transform, and load orchestrator that manages automated data movement workflows across cloud, on-premise, and hybrid environments. The platform provides a standardized interface for connectors, enabling the movement of structured and unstructured data while maintaining stateful checkpoints for reliable incremental syncing. The platform distinguishes itself through a containerized architecture that isolates connectors to prevent de

SeaTunnel is a distributed data integration engine designed to synchronize structured and unstructured data across diverse sources and sinks. It functions as a multi-engine execution framework that can run data integration tasks across different distributed computing backends to optimize workload performance. The project is distinguished by a visual data pipeline designer for configuring workflows without manual code and a specialized change data capture tool for streaming incremental database updates. It also includes an enrichment pipeline that integrates large language models and embedding

Benthos is a stream processing engine and data integration pipeline used for routing, transforming, and connecting data streams between diverse sources and sinks. It functions as event routing middleware and a change data capture tool, streaming real-time database modifications as discrete events for downstream processing. The system utilizes a declarative pipeline configuration, where data flow and processing logic are defined in a single static file. It features a specialized domain-specific language for mapping, filtering, and enriching data payloads, allowing for complex transformations w

Connect is a Kafka data integration platform and stream processing engine used to build declarative pipelines that move and transform messages between Kafka topics and external sources. It functions as a Kafka Connect framework and a change data capture tool, streaming real-time database modifications to synchronize data across distributed environments. The project differentiates itself through a dedicated mapping language for mutating and reshaping message payloads and the ability to execute custom processing logic within a sandboxed WebAssembly runtime. It also provides an observability pip

Otter is a distributed database synchronization system and change data capture tool designed to replicate data between databases across multiple geographic regions. It functions as a synchronization orchestrator and ETL data pipeline that mirrors records and associated files in real time. The system employs incremental log parsing to capture database changes and utilizes a consistency-based convergence algorithm and loop-avoidance logic to manage bi-directional replication. It processes data through a pipeline of selection, extraction, transformation, and loading to handle joins and format co

Maxwell este un instrument de captură a datelor modificate (CDC) MySQL și o aplicație de streaming binlog care convertește modificările bazei de date în evenimente JSON structurate. Acesta funcționează ca o conductă de date care citește log-urile binare MySQL pentru a sincroniza modificările între indici externi, motoare de căutare și sisteme de mesagerie distribuite, cum ar fi Kafka. Proiectul oferă capabilități pentru a menține audit trails persistente prin înregistrarea unui istoric cronologic al tuturor modificărilor bazei de date. Acesta permite sincronizarea datelor în timp real și integrarea arhitecturii bazate pe evenimente prin streaming-ul modificărilor bazei de date către platforme externe pentru a declanșa fluxuri de lucru și a notifica microserviciile. Sistemul acoperă domenii funcționale largi, inclusiv bootstrapping-ul datelor prin snapshot-uri inițiale, gestionarea versiunilor de schemă și filtrarea evenimentelor. Încorporează gestionarea traficului prin rutare bazată pe chei de partiție și oferă monitorizare prin verificări de sănătate și metrici de performanță expuse printr-un endpoint HTTP. Conexiunile la baze de date și producătorii de streaming sunt securizate folosind SSL și comunicare criptată.

Chunjun este un framework distribuit de integrare a datelor și pipeline ETL bazat pe SQL, conceput pentru a sincroniza datele între surse eterogene. Acesta funcționează ca un instrument de change data capture și un sincronizator de date eterogene, utilizând un mediu de procesare distribuit pentru a muta și transforma datele între diferite tipuri de baze de date. Sistemul se distinge prin arhitectura sa de conectori bazată pe plugin-uri, care permite dezvoltarea de plugin-uri personalizate de sursă și destinație pentru a extinde conectivitatea către sisteme de date neacceptate. Suportă change data capture în timp real din log-urile bazelor de date relaționale și implementează propagarea evoluției schemei pentru a aplica automat modificările structurale de la tabelele sursă la cele de destinație. Framework-ul oferă capabilități pentru sincronizarea incrementală a datelor și calculul datelor între surse folosind logica SQL. Fiabilitatea este gestionată prin recuperarea sarcinilor bazată pe checkpoint-uri pentru a relua transferurile întrerupte și cozi de mesaje dead-letter pentru gestionarea datelor murdare, pentru a audita înregistrările malformate. Sarcinile de integrare pot fi implementate pe clustere standalone, Yarn sau medii Kubernetes, cu suport pentru implementare containerizată prin Docker.