2 dépôts
Accelerates the identification of matching data blocks using high-performance rolling checksum algorithms.
Distinct from Algorithmic Performance Optimizations: Specifically applies to block-level difference detection for data transfer, not general time/space complexity of algorithms.
Explore 2 awesome GitHub repositories matching software engineering & architecture · Checksum-Based Block Matching. Refine with filters or upvote what's useful.
kafka-python is a pure-Python client library for Apache Kafka that implements the Kafka wire protocol directly, without any native bindings or JVM dependencies. It provides the core capabilities of a Kafka client: producing messages to topics, consuming records from topics, and administering cluster resources such as topics and partitions, all through a Pythonic API or command-line tools. The library distinguishes itself through its comprehensive support for advanced Kafka features. It includes an asynchronous producer with background batching for throughput, a consumer group rebalance protoc
Offloads CRC32C checksum computation to an optimized C extension to reduce CPU overhead during record writing.
rsync is a file synchronization and transfer tool that copies data between local and remote systems by sending only the differences between source and destination files. It computes matching blocks using checksums, so only the unmatched portions of files are transmitted, making repeated synchronizations much faster than full copies. The tool preserves file metadata such as ownership, permissions, and modification times during transfers. rsync supports multiple transfer methods, including direct connections through a dedicated daemon mode or via remote shells like SSH. It can mirror directory
Identifies matching blocks between source and destination using checksums, so only unmatched parts are transmitted.