apache/druid
Druid
Apache Druid is a real-time analytics database and distributed columnar time-series store designed for sub-second analytical queries. It functions as a data platform featuring a distributed SQL query engine and a real-time data ingestion system for moving historical and streaming data from external sources.
The system is distinguished by its ability to provide low-latency analytics under high concurrency to power operational dashboards. It implements a Kerberos-secured environment for user authentication and employs a shared-nothing cluster architecture to enable horizontal scaling.
The platform covers large-scale data transformation through multi-stage SQL processing and query planning. It includes capabilities for distributed cluster administration, infrastructure state tracking, and resource monitoring via web-based consoles.
The project provides utilities for query workload capture and SQL result validation to ensure consistency across versions.
Features
- Real-time Analytics Platforms - Functions as a distributed columnar store designed for sub-second analytical queries and high-concurrency operational dashboards.
- Columnar Storage Engines - Implements a storage engine that organizes data by column to optimize analytical workloads and reduce disk I/O.
- Multi-Stage Pipeline Processing - Employs a dedicated engine to run multi-stage query pipelines for high-performance data processing.
- Distributed Query Stages - Breaks complex SQL executions into a sequence of distributed stages to handle large-scale data transformations.
- Distributed SQL Querying - Provides a distributed engine that translates SQL into executable plans for complex data transformations across a cluster.
- Event Stores - Provides a high-performance persistence mechanism specifically designed for managing and querying event-driven datasets.
- Large-Scale Data Computation - Executes complex data analysis and multi-stage SQL transformations across distributed clusters for massive datasets.
- Low-Latency Analytical Queries - Sub-second query result delivery under high concurrency to power operational dashboards and user interfaces.
- Query Planning - Constructs and optimizes SQL execution plans to ensure efficient data retrieval across the cluster.
- Real-Time Data Streaming - Provides an integrated event-driven streaming system to capture and index data for immediate analytical availability.
- SQL Query Execution Engines - Processes standard SQL and native queries across a distributed cluster to analyze large-scale datasets.
- Columnar Time-Series Indexing - Organizes event-driven datasets in columns to optimize the performance of large-scale aggregations on time-series data.
- Time-Based Segment Partitioning - Splits data into time-based chunks distributed across the cluster to enable parallel processing and scalable retrieval.
- Shared-Nothing Processing Engines - Employs a shared-nothing processing model to execute tasks independently on worker nodes, enabling horizontal scaling.
- Analytical Workbenches - Ships a built-in workbench and connectivity toolset for prototyping and executing SQL and native queries.
- Hybrid Ingestion Wizards - Provides a guided setup wizard for loading and monitoring both real-time and historical data ingestion tasks.
- Concurrent Analytical Serving - Enables the serving of complex analytical queries to many simultaneous users across distributed clusters without performance loss.
- Native Query Translation - Converts standard SQL expressions into a proprietary native query format for optimized execution on the storage engine.
- Cluster Administration - Provides tools for managing the operational health and resource allocation of the distributed database cluster.
- Kerberos Authentication - Integrates the Kerberos protocol to provide secure user authentication within the distributed data platform.
- Administrative Interfaces - Provides a web-based dashboard for managing system configurations and operational settings across the distributed cluster.
- Cluster Monitoring - Implements system tables and visual consoles to monitor cluster health and resource allocation.
- Database Management Consoles - Includes a centralized control plane for managing and interacting with the analytics database without raw API calls.
- Infrastructure State Tracking - Enables tracking of cluster infrastructure, including datasources and segments, via specialized system tables.
- Query Performance Monitoring - Reports real-time counters from worker tasks to a central controller to monitor query progress and performance.
- Analytics - Distributed, real-time analytics data store for high-performance queries.
- Analytics Tools - Listed in the “Analytics Tools” section of the Awesome Selfhosted awesome list.
- Data Storage Systems - Delivers high-performance real-time analytics.
- API and Data Services - Analyzes real-time data with high performance.
