Debezium

Debezium is a distributed change data capture platform that streams row-level database modifications as real-time events. By parsing database transaction logs, the system broadcasts structural and data changes to message brokers, enabling reactive processing and data integration across distributed architectures.

The platform utilizes log-based capture to extract modifications directly from transaction logs, ensuring minimal impact on source system performance while maintaining the original commit order of operations. It employs database-specific connector adapters to translate proprietary binary formats into a unified event structure, supported by schema-registry-backed serialization to maintain consistent data definitions. To ensure a complete baseline for synchronization, the system performs snapshot-based initial states before transitioning to continuous event streaming.

The tool supports a broad range of data integration tasks, including the maintenance of analytical stores and the synchronization of data across operational systems. Users can refine the data stream by applying filters to include or exclude specific tables, columns, or data types, and the system maintains an accurate representation of data models by parsing structural statements during the capture process.

The project is implemented as a plugin for distributed message queues, facilitating integration into existing event-driven pipelines.

Features

Change Data Capture - Provides a distributed platform for streaming row-level database modifications as real-time events.
Event Streaming Infrastructure - Parses database transaction logs to broadcast structural and data changes to external applications.
Message Queue Connectors - Integrates as a plugin for streaming database transaction logs into distributed message queues.
Change Data Capture Streams - Streams row-level modifications from databases to message brokers for real-time data integration.
Asynchronous Log Streaming - Broadcasts database modifications as asynchronous event streams to decouple capture from downstream consumers.
Change Data Capture - Distributed platform turning database changes into real-time event streams.
Real-Time Data Integration Platforms - Synchronizes operational database changes across software architectures in real-time.
Data Warehouse Integrations - Maintains analytical stores by streaming live database updates into data warehouses for real-time intelligence.
Database Adapters - Uses modular adapters to translate proprietary database log formats into a unified event structure.
Database Snapshots - Performs snapshot-based initial state capture to ensure a consistent baseline for data synchronization.
Transaction Orderers - Maintains the original commit order of database operations to preserve data integrity during streaming.
Schema Registries - Encodes data changes using a central schema registry to maintain consistent definitions across distributed systems.

redpanda-data/connect

8,681View on GitHub

Connect is a Kafka data integration platform and stream processing engine used to build declarative pipelines that move and transform messages between Kafka topics and external sources. It functions as a Kafka Connect framework and a change data capture tool, streaming real-time database modifications to synchronize data across distributed environments. The project differentiates itself through a dedicated mapping language for mutating and reshaping message payloads and the ability to execute custom processing logic within a sandboxed WebAssembly runtime. It also provides an observability pip

rethinkdb/rethinkdb

26,996View on GitHub

RethinkDB is a distributed, document-oriented database designed to store and manage JSON-formatted data across scalable clusters. It utilizes a custom log-structured storage engine with B-Tree indexing to ensure high-performance disk I/O and data persistence. The system maintains high availability through automatic sharding and replication, employing a primary-replica voting consensus mechanism to handle node failures and ensure consistent cluster operations. A defining characteristic of the platform is its reactive changefeed engine, which allows applications to subscribe to live data update

alibaba/canal

29,697View on GitHub

Canal is a database replication middleware that performs change data capture by simulating a database replica. It monitors transaction logs to stream incremental data modifications to downstream systems in real time, acting as an event streaming infrastructure that transforms low-level binary logs into structured, consumable message streams. The project distinguishes itself through a high-throughput architecture that utilizes concurrent multi-threaded parsing and stateful log position tracking to ensure reliable data delivery. It employs a pluggable sink architecture that decouples data extra

redis/go-redis

22,159View on GitHub

This project is a feature-rich Go client library designed for interacting with Redis. It serves as a comprehensive interface for managing remote data stores, enabling developers to execute standard database commands, handle complex data structures, and perform asynchronous operations within Go applications. The library distinguishes itself through its support for advanced Redis capabilities, including connection pooling, pipelining, and transactional integrity. It provides specialized primitives for managing distributed clusters, including automated topology updates and request routing to sha

debeziumdebezium

Features