Awesome GitHub RepositoriesData Warehousing

ClickHouse is a high-performance, columnar analytical database designed for real-time query execution and large-scale data aggregation. It functions as a distributed data warehouse capable of processing petabytes of information, while also providing an embedded engine that integrates directly into applications for native query capabilities without external dependencies. The system is built to handle high-throughput ingestion and complex analytical workloads, delivering millisecond-level latency for interactive dashboards and operational monitoring. The platform distinguishes itself through ad

This project serves as a comprehensive technical reference for the architecture and design of data-intensive applications. It provides a structured analysis of the fundamental principles required to build reliable, scalable, and maintainable software systems, covering the core trade-offs inherent in modern data infrastructure. The repository explores the mechanics of distributed data management, including strategies for replication, partitioning, and achieving consensus across multiple nodes. It details the design of storage engines, indexing techniques, and transaction management models, whi

Doris is a distributed SQL data warehouse designed for high-performance analytical workloads and real-time data processing. It functions as a unified platform that integrates traditional relational warehousing with lakehouse query capabilities, allowing users to execute analytical operations directly against external data lakes without requiring data migration. The system distinguishes itself through a shared-nothing, massively parallel processing architecture that utilizes vectorized query execution and columnar storage to maintain sub-second latency. It supports dynamic schema evolution, en

Databend is a cloud-native data warehouse and OLAP database designed for large-scale analytics. It functions as a SQL-compliant engine and serverless analytics platform that separates compute from storage to allow for independent scaling. The system integrates vector database capabilities, indexing high-dimensional embeddings to enable semantic, hybrid, and full-text searches across massive datasets. It further distinguishes itself through serverless compute management that automatically scales resources based on demand and shuts them down during idle periods. The platform covers a broad set

Connect is a Kafka data integration platform and stream processing engine used to build declarative pipelines that move and transform messages between Kafka topics and external sources. It functions as a Kafka Connect framework and a change data capture tool, streaming real-time database modifications to synchronize data across distributed environments. The project differentiates itself through a dedicated mapping language for mutating and reshaping message payloads and the ability to execute custom processing logic within a sandboxed WebAssembly runtime. It also provides an observability pip

Pinot is a distributed, columnar analytical database designed for high-concurrency, low-latency query processing. It functions as a real-time OLAP datastore, enabling interactive, user-facing analytics by ingesting and querying massive datasets from both streaming and batch sources. The system architecture relies on a centralized controller for cluster coordination and a distributed segment-based storage model to ensure horizontal scalability. The platform distinguishes itself through a hybrid ingestion pipeline that unifies real-time event streams and historical batch data into a single quer

Apache Hive is a SQL-on-Hadoop data warehouse that enables querying and managing petabytes of data stored in distributed storage such as HDFS and cloud storage services. It provides a familiar SQL interface for batch analytics and reporting, supported by a core set of components including the HiveServer2 Thrift service for remote query execution, the Hive Metastore Service for central metadata management, the Hive ACID Transaction Engine for concurrent read-write operations, and the Hive LLAP Interactive Engine for low-latency analytical processing. The WebHCat REST API offers an HTTP interfac

JanusGraph is a distributed, elastically scalable graph database designed to store and query highly connected data across a cluster of machines. It supports the property graph data model with ACID consistency and integrates multi-model search capabilities including geo, numeric range, and full-text queries. The database also includes a Graph OLAP engine for running batch analytics and global graph computations on large datasets using the Hadoop framework. The project distinguishes itself through a masterless cluster architecture that eliminates single points of failure, allowing every node to

HBase ist ein verteilter, Wide-Column-NoSQL-Speicher und Big-Data-Speicher-Engine, die für spärliche Datensätze konzipiert ist. Sie fungiert als skalierbare spaltenorientierte Datenbank, die auf dem Hadoop Distributed File System aufbaut, um Echtzeit-Lese- und Schreibzugriffe auf massive Mengen strukturierter und unstrukturierter Daten zu ermöglichen. Das System agiert als sprachübergreifendes Datenbank-Gateway und bietet Konnektivität über native Remote Procedure Calls (RPC), REST- und Thrift-Schnittstellen. Es zeichnet sich durch ein Master-Worker-Koordinationsmodell aus, das horizontale Skalierung und Fehlertoleranz über einen Cluster hinweg ermöglicht. Das Projekt deckt ein breites Spektrum an Funktionen ab, einschließlich fein abgestimmter Zugriffskontrolle über Visibility-Labels auf Zellebene, plugbarer Datenkompression und serverseitiger Datenaggregation. Es unterstützt zudem Big-Data-Analytics-Workflows durch Map-Reduce-Integration und erlaubt die Ausführung benutzerdefinierter serverseitiger Logik. Die betriebliche Überwachung wird durch System-Metrik-Tracking und Plugin-basierten Metrik-Export bereitgestellt.

Side-Menu.Android ist eine wiederverwendbare UI-Komponente für Android-Anwendungen, die eine ausziehbare Navigationsleiste (Slide-out Navigation Drawer) bereitstellt. Sie wurde entwickelt, um Entwicklern dabei zu helfen, Anwendungsbereiche und Benutzeroptionen in einem strukturierten, verborgenen Panel zu organisieren, das eine saubere Schnittstelle für den primären Inhaltsbereich beibehält. Die Komponente zeichnet sich durch ihre visuelle Präsentation aus, die den Material Design-Richtlinien folgt, um eine konsistente und intuitive Benutzererfahrung zu gewährleisten. Sie verfügt über eine datengesteuerte Menühierarchie, die eine logische Gruppierung von Navigationselementen ermöglicht, und integriert flüssige kreisförmige Reveal-Animationen, um polierte visuelle Übergänge beim Öffnen oder Schließen des Menüs zu bieten. Durch die Kapselung komplexer Layout- und Interaktionslogik in eine einzige, modulare Klasse vereinfacht die Bibliothek die Implementierung der Navigation über mehrere Bildschirme hinweg. Sie unterstützt ereignisgesteuerte Übergänge, was es Entwicklern ermöglicht, Menüinteraktionen von Inhaltsaktualisierungen zu entkoppeln, um eine saubere und reaktionsschnelle Anwendungsarchitektur zu wahren.

Kylin is a distributed OLAP engine designed for executing fast SQL queries on massive datasets. It utilizes multi-dimensional data cubes to pre-calculate data aggregates, enabling sub-second response times for large-scale analytical queries and big data analytics. The system focuses on large-scale data warehousing and multi-dimensional data modeling. It allows for the organization and querying of vast amounts of structured data to support business intelligence and reporting workflows through distributed SQL querying.

tech-vault is a command-line technical interview bank and knowledge base designed for practicing engineering questions across various technical domains. It functions as a terminal-based application that stores structured study materials and interview questions as markdown files, which are then rendered directly within the system console. The project distinguishes itself through a delivery model that uses command-line argument parsing to filter content by topic or difficulty. It also includes a random selection algorithm to pick individual questions from the collection for spontaneous study se

OpenAddresses is an open-source geospatial data aggregator and directory that collects public domain and open-license address, parcel, and building datasets from governments and organizations worldwide. It functions as a global index and data warehouse for locating and distributing free geospatial records. The project operates a normalization pipeline that cleans and standardizes diverse source formats into a consistent global coordinate and attribute schema. This process includes a crowdsourced curation pipeline and programmatic quality validation to verify the spatial accuracy and formattin

cve-search is a vulnerability search engine and database manager designed to index, synchronize, and query CVE and CPE security vulnerability data. It functions as a security data warehouse that imports vulnerability feeds into a local database to enable fast, keyword-based discovery of security flaws. The project provides a web-based vulnerability browser and a programmatic JSON API for retrieving records and risk scores. It utilizes full-text indexing for vulnerability descriptions and implements an identity-verified security portal using the OpenID Connect standard for user authentication.