Awesome GitHub RepositoriesDatabase Replication Middleware

Canal is a database replication middleware that performs change data capture by simulating a database replica. It monitors transaction logs to stream incremental data modifications to downstream systems in real time, acting as an event streaming infrastructure that transforms low-level binary logs into structured, consumable message streams. The project distinguishes itself through a high-throughput architecture that utilizes concurrent multi-threaded parsing and stateful log position tracking to ensure reliable data delivery. It employs a pluggable sink architecture that decouples data extra

Electric is a Postgres data synchronization engine and replication proxy designed to enable local-first software. It replicates data from Postgres databases to client-side stores in real time using logical replication, allowing applications to maintain a local embedded database for offline access and low-latency updates. The system distinguishes itself by using shapes to filter and authorize specific subsets of database rows and columns before streaming them to clients or edge workers. It further supports multi-user collaboration by integrating a conflict-free replicated data type framework t

该项目是一个变更数据捕获 (CDC) 系统和同步层，用于将数据从 MySQL 数据库移动到 Elasticsearch 索引中。它作为一个关系型到文档的映射器，将数据库表转换为可搜索的文档，以实现实时数据集成和全文搜索。 该同步器通过支持关系数据去规范化而脱颖而出，它将一对多数据库连接转换为父子文档结构。它还允许进行分区表聚合，使用正则表达式模式将多个数据库表分组到一个搜索索引中。 该系统涵盖了全面的数据映射和转换，包括字段类型转换、模式映射和同步字段过滤。它采用基于管道的处理模型来解码和合并字段，利用基于快照的初始加载作为基准，并利用二进制日志流进行实时更新。