quickwit-oss/quickwit
Quickwit
Quickwit is a cloud-native, distributed search engine designed for observability data such as logs, traces, and metrics. It functions as an observability backend that decouples compute from storage by persisting indices directly in S3-compatible cloud object stores.
The system is distinguished by its compatibility with the Elasticsearch REST API, allowing it to integrate with existing clients and log shippers without reconfiguration. It also serves as an OpenTelemetry data indexer, ingesting technical data via the OpenTelemetry Protocol using gRPC and HTTP.
The engine utilizes a hybrid of columnar and inverted indexing to support both full-text search and analytical aggregations. Its capability surface covers multi-tenant data isolation through index partitioning, schema-flexible ingestion, and automated index lifecycle management including data retention policies. Data can be consumed from various sources, including message brokers and streaming queues.
The project provides tools for local service orchestration using containers to deploy development environments.
Features
- Cloud Native Object Storage - Provides a search engine that persists large volumes of observability data directly in cloud-native object storage.
- Full-Text Search Engines - Functions as a high-performance full-text search engine for indexing and retrieving large-scale observability data.
- Elasticsearch Compatible Engines - Implements the Elasticsearch API to function as a drop-in compatible search engine replacement.
- Hybrid Columnar-Inverted Indices - Combines full-text inverted indices with columnar storage for optimized keyword search and aggregations.
- OTLP Ingestion - Integrates the OpenTelemetry Protocol via gRPC and HTTP to streamline observability data flow.
- Distributed Search Engines - Executes complex boolean and range queries across distributed indices at a massive scale.
- Direct Object Store Querying - Retrieves search results with sub-second latency by querying indices stored directly in cloud object stores.
- Resource Scaling Strategies - Decouples indexers and searchers from storage to allow independent scaling of compute resources.
- Stateless Compute Scaling - Implements a stateless architecture that decouples indexing and searching to enable elastic scaling in cloud environments.
- Key-Based Partitioning - Divides indices into discrete partitions based on keys to isolate tenant workloads and optimize search performance.
- OpenTelemetry Protocol Indexing - Ingests and indexes metrics, logs, and traces using the OpenTelemetry Protocol via gRPC and HTTP.
- Multi-Tenancy - Provides data isolation across multiple indexes and partitions to support multi-tenant environments.
- Multi-Tenant Data Stores - Organizes data into separate indices and partitions to provide workload isolation for multiple tenants.
- Object Storage Persistence - Persists index data and metadata directly in S3-compatible cloud object storage.
- Object Store Search Engines - Functions as a search engine that decouples compute from storage by reading indices from S3-compatible object stores.
- Observability Data Indexing - Indexes large volumes of logs, traces, and metrics stored in cloud object storage for fast search and aggregation.
- Columnar-Inverted Hybrid Indexes - Utilizes a hybrid of columnar and inverted indexing to optimize both full-text search and analytical aggregations.
- Search Engine APIs - Exposes REST and gRPC interfaces for executing search queries and managing indices from external applications.
- Storage-Compute Architectures - Separates indexing and searching compute from storage to allow independent scaling.
- Large Scale Log Analysis - Enables high-performance full-text search and aggregation across massive observability datasets for troubleshooting.
- Multi-Tenant Infrastructure - Isolates telemetry data and queries into separate indices and partitions for multi-tenant observability.
- Log Search Engines - Acts as a cloud-native search engine optimized for observability logs and traces using object storage.
- OpenTelemetry Ingestion - Accepts observability data via the OpenTelemetry Protocol using gRPC and HTTP.
- Distributed Observability Platforms - Provides a scalable distributed backend for indexing and querying massive volumes of technical observability data.
- OpenTelemetry Exporters - Collects traces and logs via gRPC and HTTP in compliance with OpenTelemetry standards.
- Data Extraction & Ingestion - Parses non-JSON data and extracts structured fields from documents prior to indexing.
- Data Ingestion - Automatically consumes and indexes data streams from cloud messaging and streaming services.
- Data Retention Policies - Implements automated data lifecycle management to delete indices based on defined retention policies.
- Query Aggregations - Calculates summary statistics and aggregations over large datasets to identify technical patterns and trends.
- Schema-Agnostic Ingestion - Supports both strict predefined schemas and a schemaless approach for flexible data ingestion.
- Index Lifecycle Management - Controls index creation, updates, and deletions through a REST API utilizing automated index templates.
- Schemaless Indexing - Supports both strict schema and schemaless indexing to accommodate flexible and evolving data structures.
- Tenant-Based Partitioning - Divides indices into discrete partitions based on keys to isolate tenant workloads.
- API Compatibility Layers - Mimics existing search engine REST endpoints to ensure interoperability with existing tools.
- Index - Controls the creation, update, and deletion of indices via templates and retention policies.
- Logging And Aggregation - Cloud-native log management and analytics engine.
- Observability and Monitoring - Cloud-native search engine for log management.
- Observability and Monitoring - Cloud-native search engine optimized for log management.
