Awesome GitHub RepositoriesDistributed Query Engines

ClickHouse is a high-performance, columnar analytical database designed for real-time query execution and large-scale data aggregation. It functions as a distributed data warehouse capable of processing petabytes of information, while also providing an embedded engine that integrates directly into applications for native query capabilities without external dependencies. The system is built to handle high-throughput ingestion and complex analytical workloads, delivering millisecond-level latency for interactive dashboards and operational monitoring. The platform distinguishes itself through ad

Copyparty is a self-hosted file server that provides a browser-based interface for managing, browsing, uploading, and downloading files. It utilizes a virtual file system abstraction to map diverse storage backends and network-attached devices into a unified directory structure, allowing for consistent file access across various storage environments. The platform functions as a cloud synchronization gateway, enabling automated data backups and transfers between local storage and remote cloud providers through integration with standard command-line tools. It also serves as a distributed storag

Polars is a high-performance columnar data processing library designed for efficient analytical workflows. It functions as a structured data library that organizes information into typed columns, utilizing the Apache Arrow memory format to enable zero-copy data sharing and cache-friendly, vectorized operations. The engine is built to handle large-scale tabular datasets, providing both local and distributed analytical runtimes that scale from single-machine environments to multi-node clusters. The project distinguishes itself through a sophisticated lazy query engine that constructs abstract e

Loki is a horizontally scalable, highly available log aggregation engine designed to store and query massive volumes of unstructured log data. It functions as a distributed observability platform that correlates logs, metrics, and traces to provide comprehensive visibility into the health and performance of complex infrastructure. The system distinguishes itself through a distributed query execution model that processes large datasets in parallel across cluster nodes. It utilizes label-based stream indexing and a distributed index to map log data to specific chunks, enabling rapid retrieval w

OpenObserve is a unified observability data platform designed to ingest, store, and analyze logs, metrics, and traces. It functions as a cloud-native monitoring tool that centralizes telemetry from diverse sources, including standard collectors and cloud service providers, into a single, scalable system. By utilizing a columnar storage engine backed by object storage, the platform enables efficient long-term data retention and high-performance analytical querying. The platform distinguishes itself through deep integration with artificial intelligence, allowing users to query data using natura

Presto is a distributed SQL query engine designed for high-performance analytical processing across heterogeneous data sources. It functions as a data federation platform and massively parallel processing engine, allowing users to execute interactive queries against diverse storage systems without requiring data migration. By mapping remote metadata and structures to a unified relational namespace, it enables seamless cross-platform analysis through a standard SQL interface. The engine distinguishes itself through a pluggable connector architecture and a shared-nothing distributed processing

Thanos is a distributed metrics query engine and monitoring scalability suite designed to provide a unified interface for aggregating data from multiple Prometheus servers and clusters. It functions as a high availability monitoring backend that eliminates single points of failure by deduplicating data from replicated instances. The system enables long-term retention by persisting time-series data to cloud-native object storage, allowing for unlimited historical archiving beyond the limits of local disks. It further optimizes this storage through a downsampling and retention manager that comp

Thanos is a CNCF cloud native monitoring tool that provides a highly available and scalable extension to the Prometheus ecosystem. It functions as a global query engine, a long-term storage system, and a metric downsampler. The project enables a unified interface to aggregate and query metrics across multiple distributed clusters from a single view. It maintains historical data beyond local retention limits by persisting time-series metrics in object storage and eliminates data gaps by merging metrics from redundant server pairs. The system includes capabilities for reducing the resolution o

Apache DataFusion is an extensible, columnar SQL query engine that runs embedded within a host application without requiring a separate server process. It processes data in columnar batches using Apache Arrow for memory-efficient analytics, and can scale analytic workloads across multiple nodes for parallel execution. The engine supports both SQL and DataFrame queries through a modular, streaming architecture that allows custom operators, data sources, functions, and optimizer rules. The engine distinguishes itself through its modular extension framework, which enables building custom query e

Riot هو محرك بحث موزع وخادم فهرسة قائم على Go مصمم للفهرسة والاسترجاع بالنص الكامل. يعمل كنظام استرجاع يقوم بفرز المستندات حسب الصلة باستخدام خوارزميات ترتيب BM25، وتكرار المصطلح، وتكرار المستند العكسي. يوفر المحرك دعماً متخصصاً للغة الصينية، ويتميز بتجزئة النص المتزامنة وتعيين Pinyin الصوتي لمطابقة المدخلات الرومانية مع الأحرف. يستخدم بنية موزعة توظف تقسيم الفهرس القائم على التجزئة (hash-based sharding) لموازنة تحميل البيانات والإنتاجية عبر عقد خادم متعددة. يغطي النظام نطاقاً واسعاً من قدرات البحث، بما في ذلك تنفيذ استعلام المنطق البولياني، وتصفية القرب، وإدارة دورة حياة الفهرس في الوقت الفعلي. يحتفظ بفهرس سريع قابل للبحث في الذاكرة مع استخدام تخزين مدعوم بالقرص لاستمرارية البيانات ومتانتها. يتم تضمين أدوات مراقبة لتتبع استخدام الذاكرة والقرص و CPU عبر البيئة الموزعة.