Awesome GitHub RepositoriesDistributed Storage Engines

Elasticsearch is a distributed search engine and NoSQL document store designed for full-text search and real-time data retrieval. It functions as a RESTful data indexer and vector database, allowing for the storage and management of structured JSON documents across multiple nodes. The system distinguishes itself through its ability to serve as a log analytics platform for monitoring system health and security events. It incorporates vector search implementation using mathematical embeddings to support generative AI and augmented generation applications. The platform covers a broad range of c

TiKV is a cloud-native distributed transactional key-value store and storage engine. It provides a distributed database designed for horizontal scalability and strong consistency across a cluster of physical nodes. The system uses a Raft-based consensus mechanism to maintain data availability and state synchronization. It ensures ACID compliance for distributed transactions through a two-phase commit workflow and manages data distribution via multi-Raft sharding. The engine handles massive datasets using automated range splitting and cluster load balancing to distribute data across different

Nebula is a distributed graph database designed for storing and querying massive volumes of interconnected vertices and edges across a horizontally scalable cluster. It functions as a Kubernetes-native database and a distributed graph analytics engine, utilizing a Raft-based distributed store to ensure strong consistency and high availability. The system features an OpenCypher query engine for performing complex graph traversals and pattern matching. It distinguishes itself with a decoupled compute-storage architecture and a shared-nothing distributed design, allowing query processing and dat

3FS is a distributed file system and RDMA storage cluster designed for high-performance AI training and inference workloads. It functions as a strongly consistent storage layer that utilizes a disaggregated architecture to pool SSDs and memory resources across multiple nodes. The system provides specialized storage implementations including an AI training checkpoint store for parallel state preservation and a distributed key-value cache store for decoder layer vectors to optimize inference processing. It ensures data integrity through chain replication and apportioned query distribution. The

Dynomite ist eine verteilte Datensharding-Schicht und ein Key-Value-Speicher-Engine-Proxy. Es fungiert als Verteilungsschicht, die Daten über mehrere Knoten hinweg sharded und repliziert und so Single-Server-Datenspeicher in skalierbare Peer-to-Peer-Systeme umwandelt. Das System agiert als Multi-Datacenter-Datenreplikator, der Daten zwischen verschiedenen geografischen Standorten synchronisiert, um Ausfallsicherheit und hohe Verfügbarkeit bei Standortausfällen zu gewährleisten. Es verwaltet die Verteilung von Key-Value-Daten, um eine lineare Skalierung des Datenspeichers und redundante Speicherung zu ermöglichen. Das Projekt bietet Funktionen für Storage-Engine-Sharding und Hochverfügbarkeits-Networking. Es leitet eingehende Anfragen an lokale oder entfernte Speicher-Engines weiter, während es Kommunikationsprotokolle aufrechterhält und die Inter-Node-Kommunikation durch Verschlüsselung sichert.