grafana/tempo
Tempo
Grafana Tempo is a high-scale distributed tracing backend and columnar trace database. It serves as an observability data store that persists and queries spans and traces using OpenTelemetry standards, allowing for the analysis of request flows across microservices.
The system distinguishes itself by using an object-store based backend with columnar Parquet storage. This architecture enables efficient attribute searching and large-scale data retrieval through dedicated attribute columnization and block-based data partitioning. It includes a specialized TraceQL query engine for filtering traces based on span relationships and attribute metadata.
The platform covers broad telemetry capabilities, including observability data correlation that links traces with logs and metrics. It provides service performance monitoring by generating request rates, error rates, and duration metrics from trace spans, while offering tools for sensitive trace redaction and tenant-isolated data sharding.
The system supports the retrieval of traces via hexadecimal identifiers and the execution of searches over specified time ranges using TraceQL.
Features
- Trace Event Columnar Stores - Organizes trace data into columnar Parquet files to enable efficient attribute filtering and high-performance retrieval.
- Distributed Tracing - Tracks requests across multiple services using unique identifiers to identify performance bottlenecks and transaction failures.
- OpenTelemetry Compliance - Persists and queries spans and traces using OpenTelemetry standards and columnar formats.
- Distributed Tracing Backends - Persists trace data from services to reconstruct and visualize request flows across a distributed system.
- Trace Identifier Retrieval - Enables fetching a specific trace using its hexadecimal identifier to analyze a single request path.
- Object Storage Persistence - Persists data blocks in cloud object storage to ensure scalability and durability of distributed trace records.
- Dedicated Attribute Columnization - Promotes frequently queried attributes to their own storage columns to avoid scanning full data blocks.
- Time-Based Segment Partitioning - Segments trace data into time-bounded blocks with associated index files for faster searching and deletion.
- Observability Platforms - Serves as an observability data store that aggregates and correlates traces, logs, and metrics.
- Metric and Performance Monitors - Generates request rates, error rates, and duration metrics from trace spans to monitor system health.
- Trace-Derived Metric Aggregations - Calculates rates, counts, and quantiles over trace results to identify trends and performance issues.
- Trace Data Producers - Enables emitting span information tied to a common trace ID to track requests across distributed systems.
- Performance Tracing Platforms - Provides a high-scale platform for recording, storing, and analyzing distributed trace data to identify performance bottlenecks.
- Telemetry Correlation - Links traces with logs and metrics to provide a unified view of system telemetry during root cause analysis.
- Trace Querying - Supports searching for traces over a specified time range using the specialized TraceQL query language.
- Block Storage - Lists block information, compaction summaries, and cache statistics for managing data at the block level.
- Time-Series Block Storage Queries - Scans storage blocks within a time range for specific attribute key-value pairs.
- Columnar Storage Engines - Persists trace data using a columnar storage format to ensure data remains portable for external analysis engines.
- Attribute Filtering - Allows listing values for specific attributes and filtering results using a query language to isolate specific events.
- Span Filters - Enables searching for traces based on names, durations, and status codes using boolean logic.
- Trace Metric Aggregation - Processes trace spans through an aggregation pipeline to derive request rates, error rates, and duration metrics.
- Block-Based Indexing - Generates index files for storage blocks to optimize the searching and retrieval of specific trace attributes.
- Span-Level Query Optimization - Reduces retrieval latency and memory usage by processing individual data spans instead of full traces.
- Aggregation Pipelines - Processes spans through pipelines to calculate counts, averages, and sums for trace analysis.
- Multi-Tenant Isolation Layers - Separates storage blocks and configuration overrides by tenant identifiers to support multi-tenant environments.
- Distributed Transaction Analysis - Tracks requests across service boundaries to identify the exact location of failures within complex distributed transactions.
- Exemplar Queries - Identifies specific traces that contributed to a calculated metric value to provide context for performance spikes.
- Trace Data Redaction - Rewrites storage blocks to remove specified trace identifiers to ensure data privacy and compliance.
- Trace Linking - Associates specific trace identifiers with aggregated metric values to provide direct navigation from signals to traces.
- Telemetry Lifecycle Management - Manages the lifecycle of trace data, including indexing attributes, migrating tenant data, and redacting sensitive information.
- Relationship-Based Querying - Identifies traces where specific spans appear as children or descendants to map request flows between services.
- Trace Visualization - Generates automatic visual representations of trace data to uncover related issues without manual querying.
- Request Latency Sampling - Visualizes the path and duration of requests across multiple services to detect and analyze latency bottlenecks.
- Monitoring Backends - High-volume, distributed tracing backend for application performance.
- Monitoring and Alerting - Distributed tracing system.
