fluent/fluent-bit

Features

• Logging and Telemetry - Acts as a unified telemetry collector for logs, metrics, and traces using a pluggable architecture.
• Logging Pipelines - Implements a complete pipeline for collecting, formatting, and routing application logs to centralized monitoring services.
• OpenTelemetry Standard Integrations - Acts as a unified telemetry collector using OpenTelemetry standards for compatible log, metric, and trace processing.
• Streaming Data Cleaning & Enrichment - Modifies and enriches data in transit using real-time filters, scripts, and queries for stream analytics.
• Data Ingestion Sources - Ingests logs, metrics, traces, and blobs from a wide variety of pluggable input sources.
• Data Stream Processors - Implements a real-time stream processing engine to mutate or drop payloads using a sequence of filters.
• Log Processing - Transforms and filters collected log information at the edge before shipping it to final destinations.
• Log Storage Engines - Forwards processed log entries to scalable backends, cloud storage, or monitoring tools for analysis.
• Log Shipping - Ships processed log data to various external backends and destinations for long-term storage or analysis.
• Stream Filters - Utilizes a sequence of stream filters to mutate or drop payloads in real-time before final delivery.
• Streaming SQL Transformations - Executes SQL queries directly against live telemetry streams for real-time filtering and restructuring.
• Log Delivery Reliability - Ensures reliable log delivery using filesystem-backed buffering and stateful retry logic to prevent data loss.
• Disk-Backed Buffering - Provides filesystem-backed buffering with checksums to ensure reliable data delivery and prevent loss during outages.
• Pluggable Input-Output Architectures - Employs a modular plugin architecture for flexible ingestion from diverse sources and shipping to various backends.
• Delivery Retry Tracking - Tracks delivery status and schedules retries for failed transmissions to ensure reliable telemetry delivery.
• Cloud Native Observability - Provides a lightweight agent for monitoring the health and performance of distributed microservices in cloud-native environments.
• Multi-Destination Log Routings - Routes processed logs and metrics to multiple external services and storage backends using a pluggable architecture.
• Log Ingestion - Provides a modular system of input plugins to ingest telemetry and log data from diverse infrastructure sources.
• Cloud Log Shippers - Functions as a specialized agent routing application logs from cloud environments to observability backends.
• Log Forwarders - Functions as a lightweight log forwarder that gathers data from multiple sources and ships it to central backends.
• Telemetry Collection and Aggregation - Unifies the transport and streaming of logs, metrics, and traces across distributed environments using OpenTelemetry.
• Telemetry Collectors - Centralizes and aggregates telemetry from multiple sources across different architectures to ensure monitoring compatibility.
• OpenTelemetry - Collects and forwards telemetry data in compliance with the OpenTelemetry Protocol for distributed tracing and logging analysis.
• Tail-Based Sampling - Implements tail-based sampling to reduce trace volume by selecting specific traces after the request completes.
• Conditional Data Filters - Filters and transforms data streams based on defined conditional predicate logic.
• Filesystem Buffering - Prevents data loss during crashes by storing processed information in a filesystem buffer with checksums.
• Real-Time Data Processors - Ingests and transforms telemetry data streams in real-time using conditional logic for continuous analytics.
• Protocol Serialization Encoders - Serializes and encodes processed telemetry data into specific binary or text protocols for remote delivery.
• Real-Time Data Streaming - Processes and delivers telemetry data streams in real-time using filters and SQL transformations before final delivery.
• Asynchronous Networking - Uses non-blocking asynchronous networking architectures to maximize concurrency and throughput during data shipping.
• Backpressure and Backlog Management - Manages data backlogs using memory and filesystem buffers to ensure delivery during downstream outages.
• Low-Latency Data Transmission - Implements high-throughput, low-latency network transmission to ensure efficient telemetry data forwarding.
• Disk-Buffered Retries - Employs disk-buffered retries to track delivery state and prevent data loss during transmission failures.
• Metadata-Driven Routing - Directs telemetry streams to specific backend destinations based on metadata attached to individual records.
• Dynamic Library Loading - Supports extending core functionality by loading external shared libraries as plugins at runtime.
• Custom Output Destinations - Allows the implementation of custom output destinations and formatting via shared library plugins.
• Distributed Tracing - Acts as a distributed tracing agent that samples and forwards trace data to analyze performance and execution flow.
• Data Collection Agents - Fast, lightweight processor for logs and metrics.
• Logging And Aggregation - Lightweight log and data forwarder.
• Embedded Systems Tools - Data collector for embedded and server operating systems.

Star history