Awesome GitHub RepositoriesLog Processing Engines

Canal is a database replication middleware that performs change data capture by simulating a database replica. It monitors transaction logs to stream incremental data modifications to downstream systems in real time, acting as an event streaming infrastructure that transforms low-level binary logs into structured, consumable message streams. The project distinguishes itself through a high-throughput architecture that utilizes concurrent multi-threaded parsing and stateful log position tracking to ensure reliable data delivery. It employs a pluggable sink architecture that decouples data extra

This project serves as a comprehensive technical reference for the architecture and design of data-intensive applications. It provides a structured analysis of the fundamental principles required to build reliable, scalable, and maintainable software systems, covering the core trade-offs inherent in modern data infrastructure. The repository explores the mechanics of distributed data management, including strategies for replication, partitioning, and achieving consensus across multiple nodes. It details the design of storage engines, indexing techniques, and transaction management models, whi

NeDB is a JavaScript embedded NoSQL document store designed for Node.js and the browser. It functions as an in-memory data store with the option to persist documents to a local file system, ensuring data survives application restarts. The project utilizes a MongoDB-compatible API to perform data operations, allowing it to serve as a lightweight document indexing system and a persistent file database without requiring a separate database server. Capabilities include querying, inserting, updating, and deleting documents, as well as the ability to create indexes on specific fields to accelerate

ToyDB is a distributed SQL database that provides a system for storing and querying data across multiple nodes. It focuses on maintaining strong consistency and fault tolerance through the implementation of a distributed consensus algorithm. The project distinguishes itself by supporting historical data versioning, enabling time-travel queries to retrieve the state of the database from a specific point in the past. It utilizes multi-version concurrency control to manage ACID transactions and ensure data integrity during concurrent operations. The system covers relational data modeling with t

Acest proiect este o traducere în limba chineză a lucrării de cercetare originale care detaliază protocolul de consens Raft. Servește drept traducere de cercetare tehnică și ghid pentru protocolul de consens, făcând specificațiile algoritmului Raft accesibile vorbitorilor de chineză. Documentația acoperă mecanismele de bază ale sistemelor distribuite, inclusiv alegerea liderului, replicarea jurnalului și protocoalele de siguranță. Oferă o explicație detaliată a modului de menținere a unei singure surse de adevăr pe mai multe servere pentru a obține gestionarea clusterelor cu toleranță la erori. Materialul abordează replicarea mașinii de stare distribuită și gestionarea jurnalelor. Acoperă concepte tehnice precum urmărirea disponibilității bazată pe heartbeat, alegerile cu timeout randomizat, versionarea consistenței bazată pe termeni și compactarea jurnalului bazată pe snapshot-uri.

m3 is a distributed time series database designed for high-resolution metrics and high-cardinality data management. It functions as a scalable storage system and a multi-cluster query engine, providing a distributed metrics aggregator capable of downsampling and summarizing data before it is committed to storage. The project distinguishes itself through a coordinated cluster model using etcd for node membership and shard placement. It supports multiple ingestion protocols, including the Prometheus remote write protocol, InfluxDB line protocol, and Graphite Carbon plaintext protocol, and provi

braft is an embeddable C++ library that implements the Raft consensus algorithm, providing a distributed consensus engine for building fault-tolerant, replicated state machines. At its core, it manages leader election, log replication, cluster membership changes, and state machine synchronization across a cluster of nodes, ensuring strong consistency and data durability even in the face of node failures. The library distinguishes itself through a comprehensive set of mechanisms for reliable distributed coordination. It uses a randomized timeout-based leader election process with term manageme

sofa-jraft is a Java implementation of the Raft consensus algorithm. It serves as a distributed consensus engine and linearizable state machine designed to ensure high availability and data consistency across a cluster of nodes. The project provides a replicated key-value store and a coordination engine for managing distributed state. It distinguishes itself through support for multi-group consensus sharding to distribute traffic and a service provider interface that allows for custom log storage and entry encoding implementations. The system covers a wide range of distributed capabilities,

NutsDB is an ACID-compliant, embedded transactional storage engine that functions as both a disk-backed key-value store and an in-memory data structure store. It provides atomic and serializable transactions with commit and rollback capabilities to ensure strict data consistency for applications requiring a lightweight persistence layer. The engine distinguishes itself by supporting a variety of complex data types, including lists, sets, and sorted sets, alongside standard byte-slice storage. It implements a transactional storage model featuring hot backups and a compaction algorithm to maint