Awesome GitHub RepositoriesDistributed Query Processing

RethinkDB is a distributed, document-oriented database designed to store and manage JSON-formatted data across scalable clusters. It utilizes a custom log-structured storage engine with B-Tree indexing to ensure high-performance disk I/O and data persistence. The system maintains high availability through automatic sharding and replication, employing a primary-replica voting consensus mechanism to handle node failures and ensure consistent cluster operations. A defining characteristic of the platform is its reactive changefeed engine, which allows applications to subscribe to live data update

Dgraph is a distributed graph database designed to store and query highly connected data. It organizes information as nodes and edges to represent complex relationships between entities, providing a platform for managing and analyzing deeply linked datasets. The system functions as a horizontally scalable cluster that partitions data across multiple nodes to maintain performance and availability as information volume increases. It utilizes a specialized query language built for low-latency navigation of interconnected data points, allowing for the execution of complex queries across large-sca

Vitess is a database clustering system for horizontal scaling of MySQL. It functions as a middleware layer that abstracts complex sharding and physical topology, allowing applications to interact with a distributed database environment through a unified interface. By intercepting and routing SQL queries across multiple shards, it enables large-scale data management while maintaining the appearance of a single database instance. The platform distinguishes itself through its ability to perform online schema migrations and distributed transaction coordination without requiring application downti

Presto is a distributed SQL query engine designed for high-performance analytical processing across heterogeneous data sources. It functions as a data federation platform and massively parallel processing engine, allowing users to execute interactive queries against diverse storage systems without requiring data migration. By mapping remote metadata and structures to a unified relational namespace, it enables seamless cross-platform analysis through a standard SQL interface. The engine distinguishes itself through a pluggable connector architecture and a shared-nothing distributed processing

Doris is a distributed SQL data warehouse designed for high-performance analytical workloads and real-time data processing. It functions as a unified platform that integrates traditional relational warehousing with lakehouse query capabilities, allowing users to execute analytical operations directly against external data lakes without requiring data migration. The system distinguishes itself through a shared-nothing, massively parallel processing architecture that utilizes vectorized query execution and columnar storage to maintain sub-second latency. It supports dynamic schema evolution, en

Thanos is a distributed metrics query engine and monitoring scalability suite designed to provide a unified interface for aggregating data from multiple Prometheus servers and clusters. It functions as a high availability monitoring backend that eliminates single points of failure by deduplicating data from replicated instances. The system enables long-term retention by persisting time-series data to cloud-native object storage, allowing for unlimited historical archiving beyond the limits of local disks. It further optimizes this storage through a downsampling and retention manager that comp

Citus is a PostgreSQL extension that transforms a standard database into a distributed system. It functions as a sharding framework and distributed SQL engine, enabling horizontal scaling by partitioning tables across a cluster of nodes. By utilizing a coordinator-worker topology, the system manages metadata and routes queries to the appropriate nodes, allowing for parallel execution of complex operations across distributed data shards. The platform distinguishes itself through its specialized support for multi-tenant architectures and real-time analytical processing. It enables tenant-based

Nebula is a distributed graph database designed for storing and querying massive volumes of interconnected vertices and edges across a horizontally scalable cluster. It functions as a Kubernetes-native database and a distributed graph analytics engine, utilizing a Raft-based distributed store to ensure strong consistency and high availability. The system features an OpenCypher query engine for performing complex graph traversals and pattern matching. It distinguishes itself with a decoupled compute-storage architecture and a shared-nothing distributed design, allowing query processing and dat

This project is a comprehensive learning resource and reference guide for software architecture and distributed systems design. It serves as a structured curriculum for engineers to study fundamental architectural patterns, scalability strategies, and distributed computing theory, specifically tailored to prepare for technical interviews and professional engineering roles. The repository distinguishes itself by providing a curated collection of industry-standard infrastructure tools and methodologies. It covers the selection and implementation of technologies for data storage, message brokeri

Apache DataFusion is an extensible, columnar SQL query engine that runs embedded within a host application without requiring a separate server process. It processes data in columnar batches using Apache Arrow for memory-efficient analytics, and can scale analytic workloads across multiple nodes for parallel execution. The engine supports both SQL and DataFrame queries through a modular, streaming architecture that allows custom operators, data sources, functions, and optimizer rules. The engine distinguishes itself through its modular extension framework, which enables building custom query e

go-ibax is a blockchain protocol platform and decentralized application infrastructure used to deploy networks with custom governance and token economics. It provides a foundation for building decentralized applications through a framework that integrates identity management and on-chain data storage. The project features a multilingual virtual machine capable of executing smart contracts written in Go, Rust, and Solidity. It implements a sharded blockchain network to increase throughput and a privacy layer utilizing zero-knowledge proofs and homomorphic encryption to anonymize transaction da

Pinot is a distributed, columnar analytical database designed for high-concurrency, low-latency query processing. It functions as a real-time OLAP datastore, enabling interactive, user-facing analytics by ingesting and querying massive datasets from both streaming and batch sources. The system architecture relies on a centralized controller for cluster coordination and a distributed segment-based storage model to ensure horizontal scalability. The platform distinguishes itself through a hybrid ingestion pipeline that unifies real-time event streams and historical batch data into a single quer

GreptimeDB is a distributed, open-source time-series database built for unified observability. It stores and queries metrics, logs, and traces together in a single columnar engine, supporting both SQL and PromQL for analysis. The database is designed as a Kubernetes-native operator with a decoupled compute and storage architecture, enabling horizontal scaling and multi-region deployment. What distinguishes GreptimeDB is its role as a multi-protocol ingestion gateway, accepting data through OpenTelemetry, Prometheus Remote Write, InfluxDB, Loki, Elasticsearch, Kafka, and MQTT protocols without

Mimir este o bază de date de serii temporale multi-tenant și un magazin de metrici distribuit, conceput pentru telemetrie scalabilă. Servește ca un backend compatibil cu Prometheus, oferind stocare pe termen lung și un motor de interogare scalabil pentru volume masive de date de serii temporale. Sistemul este construit pentru observabilitate multi-tenant, izolând datele de telemetrie și limitele de resurse pentru echipe sau organizații independente în cadrul unui singur cluster. Asigură disponibilitate ridicată și durabilitate prin sharding și replicarea datelor într-un cluster distribuit, utilizând stocarea de obiecte pentru persistență pentru a elimina dependențele de baze de date externe. Proiectul acoperă capabilități vaste, inclusiv agregarea globală a metricilor pentru analiză cross-region și execuția distribuită a interogărilor folosind paralelizarea și caching-ul. De asemenea, integrează instrumente de observabilitate precum alertarea federată, monitorizarea sintetică și fluxuri de lucru de rezolvare a incidentelor bazate pe AI pentru a accelera depanarea. Controalele administrative includ cote de resurse pentru chiriași, override-uri de resurse per-utilizator și shuffle-sharding pentru izolarea sarcinilor de lucru.

OpenTSDB este o bază de date distribuită de serii temporale și un motor de metrici conceput pentru stocarea și gestionarea unor volume masive de metrici de sistem cu cardinalitate ridicată. Acesta funcționează ca un depozit de date și o platformă de analiză care permite ingestia de metrici la scară largă și monitorizarea performanței infrastructurii într-un cluster distribuit. Sistemul se distinge printr-o abstractizare a stocării distribuite care suportă mai multe backend-uri, cum ar fi HBase, Cassandra și Google Bigtable. Utilizează un arbore ierarhic de metrici pentru a organiza seriile temporale și folosește indexarea cu identificatori numerici pentru a reduce amprenta de stocare și a accelera căutările pentru metricile etichetate. Proiectul acoperă domenii largi de capabilități, inclusiv analiza datelor de serii temporale cu calcule distribuite de percentile și downsampling, precum și gestionarea cuprinzătoare a metadatelor. Oferă integrare API pentru ingestia și interogarea datelor, caching off-heap pentru optimizarea performanței și instrumente pentru auditarea integrității datelor și analiza anomaliilor. Sistemul este gestionat printr-o interfață linie de comandă pentru administrarea bazei de date și sincronizarea arborelui de metrici.

m3 is a distributed time series database designed for high-resolution metrics and high-cardinality data management. It functions as a scalable storage system and a multi-cluster query engine, providing a distributed metrics aggregator capable of downsampling and summarizing data before it is committed to storage. The project distinguishes itself through a coordinated cluster model using etcd for node membership and shard placement. It supports multiple ingestion protocols, including the Prometheus remote write protocol, InfluxDB line protocol, and Graphite Carbon plaintext protocol, and provi

YDB este o bază de date SQL distribuită și un motor analitic conceput pentru scalabilitate orizontală și consistență puternică. Funcționează ca un sistem multi-model care suportă workload-uri tranzacționale și analitice printr-o arhitectură distribuită care oferă tranzacții ACID serializabile. Sistemul se distinge prin compatibilitatea sa largă cu protocoalele, implementând protocolul PostgreSQL pentru driverele SQL standard și protocolul Kafka pentru mesagerie și streaming. Servește, de asemenea, ca o bază de date vectorială, suportând indecși vectoriali și căutări de tip approximate nearest neighbor pentru căutări semantice și embeddings. Platforma gestionează datele folosind un model de stocare hibrid cu formate orientate pe rânduri și pe coloane, utilizând execuția interogărilor vectorizate pentru analize la scară de petabytes. Suprafața sa operațională include streaming de tip change data capture, cozi persistente de tip exactly-once și disponibilitate ridicată multi-zonă. Deployment-ul și gestionarea ciclului de viață sunt susținute printr-un operator Kubernetes și provizionarea de tip infrastructure-as-code.

Olric is a distributed data grid and in-memory key-value store that partitions and replicates data across a cluster of servers. It serves as a shared memory system for managing distributed maps, performing atomic operations, and acting as an in-memory data cache. The system provides a distributed locking mechanism for concurrency control and a pub-sub messaging system that broadcasts and routes messages over named channels across the cluster. The platform covers wide-ranging capabilities including cluster management and orchestration, data replication with configurable quorums, and automated