Awesome GitHub RepositoriesData Lake Acceleration

ClickHouse is a high-performance, columnar analytical database designed for real-time query execution and large-scale data aggregation. It functions as a distributed data warehouse capable of processing petabytes of information, while also providing an embedded engine that integrates directly into applications for native query capabilities without external dependencies. The system is built to handle high-throughput ingestion and complex analytical workloads, delivering millisecond-level latency for interactive dashboards and operational monitoring. The platform distinguishes itself through ad

Presto is a distributed SQL query engine designed for high-performance analytical processing across heterogeneous data sources. It functions as a data federation platform and massively parallel processing engine, allowing users to execute interactive queries against diverse storage systems without requiring data migration. By mapping remote metadata and structures to a unified relational namespace, it enables seamless cross-platform analysis through a standard SQL interface. The engine distinguishes itself through a pluggable connector architecture and a shared-nothing distributed processing

Doris is a distributed SQL data warehouse designed for high-performance analytical workloads and real-time data processing. It functions as a unified platform that integrates traditional relational warehousing with lakehouse query capabilities, allowing users to execute analytical operations directly against external data lakes without requiring data migration. The system distinguishes itself through a shared-nothing, massively parallel processing architecture that utilizes vectorized query execution and columnar storage to maintain sub-second latency. It supports dynamic schema evolution, en

Apache Hudi is an open-source table format that brings ACID transactions, incremental processing, and multi-modal indexing to data lakes. It provides atomic commits with snapshot isolation, rollback, and optimistic concurrency control for reliable data lake operations, while supporting upserts, record-level updates, and deletions in large analytical datasets. The project distinguishes itself through a timeline-based architecture that coordinates all write operations, enabling features like time-travel querying, incremental change streaming, and multi-modal query views that include snapshot, i

lakeFS es un sistema de versionado de lagos de datos que proporciona ramificaciones (branching) y commits similares a Git para grandes conjuntos de datos almacenados en almacenamiento de objetos. Funciona como una capa de control de versiones, permitiendo la creación de instantáneas inmutables, commits atómicos y ramificaciones de copia cero para crear entornos aislados para la experimentación de datos sin duplicar archivos físicos. El sistema sirve como una puerta de enlace de almacenamiento compatible con S3 y un catálogo REST de Iceberg, permitiendo que los protocolos de almacenamiento en la nube estándar y los clientes compatibles gestionen tablas versionadas. Actúa como un guardián de calidad de datos mediante el uso de un sistema de hooks basado en eventos para validar conjuntos de datos contra políticas de gobernanza antes de que los cambios se fusionen en producción. La plataforma cubre amplias capacidades para la gobernanza de datos, incluyendo colaboración mediante pull requests, control de acceso basado en roles y seguimiento del linaje de datos. Proporciona integración para la orquestación de flujos de trabajo, pipelines de aprendizaje automático y varios motores de cómputo de big data, soportando conectividad de almacenamiento multi-nube y sincronización de identidad mediante SSO y SCIM. El software se puede instalar utilizando binarios, contenedores o Helm charts para su despliegue en Kubernetes.

MiniOB is an open-source educational relational database kernel designed for learning the internals of database systems. It implements a dual-engine storage architecture combining B+ Tree and LSM-Tree, supports SQL parsing and query execution, and provides transactional processing with multi-version concurrency control. The system communicates with clients using the MySQL wire protocol and includes a vector database extension for storing and querying high-dimensional vectors. The project distinguishes itself through its comprehensive coverage of core database concepts in a single, learnable c