fluent/fluentd

Features

• Monitoring and Logging - Aggregates logs from diverse sources and routes them to central storage backends for unified analysis.
• Log Routing - Gathers log information from diverse sources, parses it, and routes it to multiple storage backends.
• Data Pipeline Configurations - Defines data pipelines using a structured configuration format supporting YAML and a domain-specific language.
• Distributed Event Routers - Directs data streams to specific output destinations by matching event labels against regular expression patterns.
• Log Aggregators - Functions as a unified logging layer that centralizes logs from distributed sources into unified stores.
• Log Storage Engines - Routes gathered log events to various storage backends, including databases, cloud services, and files.
• Record Transformers - Modifies event content by parsing fields, filtering records via grep, or changing record structures.
• Unstructured Data Transformation Tools - Transforms raw unstructured log strings into structured data formats using regular expressions and specialized parsers.
• Tag-Based Routing - Directs log events to specific output destinations using regular expression matches against event tags.
• Plugin-Based Architectures - Implements a modular architecture where data flows through a sequence of independent input, filter, and output plugins.
• Telemetry and Log Collectors - Receives and captures log data via HTTP, TCP, and UDP sockets for centralized processing.
• Log Transformation Pipelines - Modifies and enriches log entries during ingestion to transform raw strings into structured, searchable formats.
• Log Parsers - Transforms raw log strings into structured formats using regular expressions, JSON, and syslog parsers.
• Log Processing Pipelines - Orchestrates the parsing, normalizing, and structuring of raw log data using customizable pipelines.
• Label-Based Routing - Directs log data streams to specific output destinations based on event labels and regular expression patterns.
• Log Forwarders - Acts as a lightweight agent that monitors and transmits logs from local disks or sockets to remote endpoints.
• Observability Pipelines - Builds reliable data pipelines that buffer and filter logs to ensure delivery to monitoring platforms.
• Syslog Ingestion - Processes system log messages following RFC3164 and RFC5424 formatting standards.
• Inode-Based Tracking - Tracks log files via unique filesystem identifiers to ensure continuous reading across file rotations and renames.
• Event Buffering - Provides mechanisms to temporarily store event streams in memory or on disk to prevent data loss during spikes.
• Process Scaling - Distributes processing load across multiple independent processes to maximize throughput and enable zero-downtime restarts.
• Pipeline Plugins - Implements a pipeline architecture where data flows through a sequence of independent input, filter, and output plugins.
• HTTP Log Ingestion - Receives log data via HTTP requests using JSON, msgpack, or NDJSON payloads.
• Log File Tailing - Tails log files from disk and emits new lines as events using glob patterns and inode tracking.
• Internal Telemetry Monitors - Tracks internal operational state, including input throughput, buffer usage, and dropped event counts.
• Process Lifecycle Managers - Manages the lifecycle of worker processes to enable zero-downtime restarts and scaling.
• Remote Log Transmission - Pushes log data to remote web endpoints using connection reuse and secure request signing.
• TCP/UDP Log Ingestion - Listens for incoming log data on TCP or UDP sockets with configurable message length limits.
• Data Collection Agents - Unified logging layer for collecting and consuming data.
• Log Management - Unified logging layer for data collection.
• Log Management - Unified logging layer for data collection.
• Observability and Monitoring - Unified logging layer for data collection.
• Logging - Collects and routes events from various data sources.

Star history