4 个仓库
Awesome GitHub RepositoriesBatch Write Buffering
Grouping multiple data records into single transactions to increase ingestion throughput and reduce network overhead.
Distinct from Multi-Table Batch Writes: General-purpose batching for throughput, unlike the candidates which focus on LSM-trees, multi-table writes, or offline feature stores.
Explore 4 awesome GitHub repositories matching data & databases · Batch Write Buffering. Refine with filters or upvote what's useful.
Awesome Batch Write Buffering GitHub Repositories
用 AI 发现最棒的仓库。我们将通过 AI 为您搜索最匹配的仓库。
- alibaba/datax
alibaba/DataX
17,241在 GitHub 上查看DataX is a distributed data integration framework and plugin-based ETL tool designed for synchronizing large datasets between heterogeneous sources and destinations. It functions as a JDBC data migration engine and offline synchronization tool, enabling the movement of data between relational databases, NoSQL stores, and object storage. The system utilizes a plugin-based connector architecture that decouples reader and writer logic, allowing it to map and transform data types across different storage engines using a standardized internal representation. This design supports heterogeneous data
Groups multiple record writes into a single transaction to increase data ingestion speed and reduce network overhead.
Java - jitsucom/jitsu
jitsucom/jitsu
4,782在 GitHub 上查看Jitsu 是一个客户数据平台，旨在收集、转换并将应用程序事件路由到数据仓库和营销工具。它作为一个事件摄取引擎和数据仓库路由器，通过 API 和 SDK 捕获行为数据以进行实时处理和存储。 该平台具有可编程的 JavaScript 数据流水线，允许在传输过程中对事件数据进行过滤、丰富和重塑。它包含一个客户身份拼接器，可合并匿名和已知的用户标识符，以在仓库中维护持久的客户画像。 系统涵盖了广泛的功能，包括来自 Web 和移动环境的多源事件收集、目标仓库的自动模式演进，以及到 SaaS 平台和 SQL 数据库的多目标路由。它为开发人员提供了一套用于测试转换逻辑的工具，并支持通过 Kubernetes 或自托管 Docker 环境进行部署。
Buffers individual events into temporary memory structures to optimize network throughput and minimize write operations to target databases.
TypeScriptbigqueryclickhousedata-collection - zendesk/maxwell
zendesk/maxwell
4,254在 GitHub 上查看Maxwell 是一个 MySQL 变更数据捕获 (CDC) 工具和 binlog 流处理应用程序，可将数据库修改转换为结构化 JSON 事件。它作为一个数据管道，读取 MySQL 二进制日志，以便在外部索引、搜索引擎和 Kafka 等分布式消息系统中同步变更。 该项目提供了通过记录所有数据库修改的时间顺序历史来维护持久审计跟踪的功能。它通过将数据库变更流式传输到外部平台来触发工作流并通知微服务，从而实现实时数据同步和事件驱动架构集成。 该系统涵盖了广泛的功能领域，包括通过初始快照进行数据引导、模式版本管理和事件过滤。它通过分区键路由进行流量管理，并提供通过 HTTP 端点暴露的健康检查和性能指标监控。 与数据库和流处理生产者的连接使用 SSL 和加密通信进行保护。
Implements batch write buffering to increase throughput and reduce network overhead when streaming database changes.
Java - sofastack/sofa-jraft
sofastack/sofa-jraft
3,806在 GitHub 上查看sofa-jraft is a Java implementation of the Raft consensus algorithm. It serves as a distributed consensus engine and linearizable state machine designed to ensure high availability and data consistency across a cluster of nodes. The project provides a replicated key-value store and a coordination engine for managing distributed state. It distinguishes itself through support for multi-group consensus sharding to distribute traffic and a service provider interface that allows for custom log storage and entry encoding implementations. The system covers a wide range of distributed capabilities,
Supports batch write operations across different partitions in parallel to increase data throughput.
Javaconsensusdistributed-consensus-algorithmsjava
