fluent/fluentd
Fluentd
Fluentd is a unified logging layer and distributed event router that collects, parses, and routes log data from diverse sources to various storage backends. It functions as a log forwarding agent and pipeline orchestrator, transforming raw unstructured log strings into formatted objects using structured log parsing.
The project utilizes a plugin-based pipeline architecture to route data through independent input, filter, and output stages. It differentiates itself through tag-based event routing, which uses regular expression patterns to direct specific data streams to their intended destinations.
Its capabilities cover wide-ranging data collection via HTTP, TCP, UDP, and local file tailing, as well as the processing of system logs following RFC standards. The system includes internal monitoring for throughput and buffer usage, as well as mechanisms for event buffering and compression to prevent data loss during traffic spikes or output failures.
Pipelines are defined using a structured configuration format supporting YAML or a domain-specific language.
Features
- Monitoring and Logging - Aggregates logs from diverse sources and routes them to central storage backends for unified analysis.
- Log Routing - Gathers log information from diverse sources, parses it, and routes it to multiple storage backends.
- Data Pipeline Configurations - Defines data pipelines using a structured configuration format supporting YAML and a domain-specific language.
- Distributed Event Routers - Directs data streams to specific output destinations by matching event labels against regular expression patterns.
- Log Aggregators - Functions as a unified logging layer that centralizes logs from distributed sources into unified stores.
- Log Storage Engines - Routes gathered log events to various storage backends, including databases, cloud services, and files.
- Record Transformers - Modifies event content by parsing fields, filtering records via grep, or changing record structures.
- Unstructured Data Transformation Tools - Transforms raw unstructured log strings into structured data formats using regular expressions and specialized parsers.
- Tag-Based Routing - Directs log events to specific output destinations using regular expression matches against event tags.
- Plugin-Based Architectures - Implements a modular architecture where data flows through a sequence of independent input, filter, and output plugins.
- Telemetry and Log Collectors - Receives and captures log data via HTTP, TCP, and UDP sockets for centralized processing.
- Log Transformation Pipelines - Modifies and enriches log entries during ingestion to transform raw strings into structured, searchable formats.
- Log Parsers - Transforms raw log strings into structured formats using regular expressions, JSON, and syslog parsers.
- Log Processing Pipelines - Orchestrates the parsing, normalizing, and structuring of raw log data using customizable pipelines.
- Label-Based Routing - Directs log data streams to specific output destinations based on event labels and regular expression patterns.
- Log Forwarders - Acts as a lightweight agent that monitors and transmits logs from local disks or sockets to remote endpoints.
- Observability Pipelines - Builds reliable data pipelines that buffer and filter logs to ensure delivery to monitoring platforms.
- Syslog Ingestion - Processes system log messages following RFC3164 and RFC5424 formatting standards.
- Inode-Based Tracking - Tracks log files via unique filesystem identifiers to ensure continuous reading across file rotations and renames.
- Event Buffering - Provides mechanisms to temporarily store event streams in memory or on disk to prevent data loss during spikes.
- Process Scaling - Distributes processing load across multiple independent processes to maximize throughput and enable zero-downtime restarts.
- Pipeline Plugins - Implements a pipeline architecture where data flows through a sequence of independent input, filter, and output plugins.
- HTTP Log Ingestion - Receives log data via HTTP requests using JSON, msgpack, or NDJSON payloads.
- Log File Tailing - Tails log files from disk and emits new lines as events using glob patterns and inode tracking.
- Internal Telemetry Monitors - Tracks internal operational state, including input throughput, buffer usage, and dropped event counts.
- Process Lifecycle Managers - Manages the lifecycle of worker processes to enable zero-downtime restarts and scaling.
- Remote Log Transmission - Pushes log data to remote web endpoints using connection reuse and secure request signing.
- TCP/UDP Log Ingestion - Listens for incoming log data on TCP or UDP sockets with configurable message length limits.
- Data Collection Agents - Unified logging layer for collecting and consuming data.
- Log Management - Unified logging layer for data collection.
- Logging And Aggregation - Unified logging layer for cloud-native systems.
- Log Management - Unified logging layer for data collection.
- Observability and Monitoring - Unified logging layer for data collection.
- Logging - Collects and routes events from various data sources.
