Awesome GitHub RepositoriesDistributed Query Engines

ClickHouse is a high-performance, columnar analytical database designed for real-time query execution and large-scale data aggregation. It functions as a distributed data warehouse capable of processing petabytes of information, while also providing an embedded engine that integrates directly into applications for native query capabilities without external dependencies. The system is built to handle high-throughput ingestion and complex analytical workloads, delivering millisecond-level latency for interactive dashboards and operational monitoring. The platform distinguishes itself through ad

Copyparty is a self-hosted file server that provides a browser-based interface for managing, browsing, uploading, and downloading files. It utilizes a virtual file system abstraction to map diverse storage backends and network-attached devices into a unified directory structure, allowing for consistent file access across various storage environments. The platform functions as a cloud synchronization gateway, enabling automated data backups and transfers between local storage and remote cloud providers through integration with standard command-line tools. It also serves as a distributed storag

Polars is a high-performance columnar data processing library designed for efficient analytical workflows. It functions as a structured data library that organizes information into typed columns, utilizing the Apache Arrow memory format to enable zero-copy data sharing and cache-friendly, vectorized operations. The engine is built to handle large-scale tabular datasets, providing both local and distributed analytical runtimes that scale from single-machine environments to multi-node clusters. The project distinguishes itself through a sophisticated lazy query engine that constructs abstract e

Loki is a horizontally scalable, highly available log aggregation engine designed to store and query massive volumes of unstructured log data. It functions as a distributed observability platform that correlates logs, metrics, and traces to provide comprehensive visibility into the health and performance of complex infrastructure. The system distinguishes itself through a distributed query execution model that processes large datasets in parallel across cluster nodes. It utilizes label-based stream indexing and a distributed index to map log data to specific chunks, enabling rapid retrieval w

OpenObserve is a unified observability data platform designed to ingest, store, and analyze logs, metrics, and traces. It functions as a cloud-native monitoring tool that centralizes telemetry from diverse sources, including standard collectors and cloud service providers, into a single, scalable system. By utilizing a columnar storage engine backed by object storage, the platform enables efficient long-term data retention and high-performance analytical querying. The platform distinguishes itself through deep integration with artificial intelligence, allowing users to query data using natura

Presto is a distributed SQL query engine designed for high-performance analytical processing across heterogeneous data sources. It functions as a data federation platform and massively parallel processing engine, allowing users to execute interactive queries against diverse storage systems without requiring data migration. By mapping remote metadata and structures to a unified relational namespace, it enables seamless cross-platform analysis through a standard SQL interface. The engine distinguishes itself through a pluggable connector architecture and a shared-nothing distributed processing

Thanos is a distributed metrics query engine and monitoring scalability suite designed to provide a unified interface for aggregating data from multiple Prometheus servers and clusters. It functions as a high availability monitoring backend that eliminates single points of failure by deduplicating data from replicated instances. The system enables long-term retention by persisting time-series data to cloud-native object storage, allowing for unlimited historical archiving beyond the limits of local disks. It further optimizes this storage through a downsampling and retention manager that comp

Thanos is a CNCF cloud native monitoring tool that provides a highly available and scalable extension to the Prometheus ecosystem. It functions as a global query engine, a long-term storage system, and a metric downsampler. The project enables a unified interface to aggregate and query metrics across multiple distributed clusters from a single view. It maintains historical data beyond local retention limits by persisting time-series metrics in object storage and eliminates data gaps by merging metrics from redundant server pairs. The system includes capabilities for reducing the resolution o

Apache DataFusion is an extensible, columnar SQL query engine that runs embedded within a host application without requiring a separate server process. It processes data in columnar batches using Apache Arrow for memory-efficient analytics, and can scale analytic workloads across multiple nodes for parallel execution. The engine supports both SQL and DataFrame queries through a modular, streaming architecture that allows custom operators, data sources, functions, and optimizer rules. The engine distinguishes itself through its modular extension framework, which enables building custom query e

Riot es un motor de búsqueda distribuido y servidor de indexación basado en Go diseñado para indexación y recuperación de texto completo. Funciona como un sistema de recuperación que ordena documentos por relevancia utilizando algoritmos de ranking BM25, frecuencia de términos y frecuencia inversa de documento (TF-IDF). El motor proporciona soporte especializado para el idioma chino, con segmentación de texto concurrente y mapeo fonético Pinyin para hacer coincidir la entrada romanizada con los caracteres. Utiliza una arquitectura distribuida que emplea fragmentación (sharding) de índice basada en hash para equilibrar la carga de datos y el rendimiento a través de múltiples nodos de servidor. El sistema cubre una amplia gama de capacidades de búsqueda, incluyendo ejecución de consultas con lógica booleana, filtrado de proximidad y gestión del ciclo de vida del índice en tiempo real. Mantiene un índice de búsqueda rápido en memoria mientras utiliza almacenamiento respaldado por disco para la persistencia y durabilidad de los datos. Se incluyen herramientas de monitoreo para rastrear la utilización de memoria, disco y CPU en todo el entorno distribuido.