Fluentd is a unified logging layer and distributed event router that collects, parses, and routes log data from diverse sources to various storage backends. It functions as a log forwarding agent and pipeline orchestrator, transforming raw unstructured log strings into formatted objects using structured log parsing.
The project utilizes a plugin-based pipeline architecture to route data through independent input, filter, and output stages. It differentiates itself through tag-based event routing, which uses regular expression patterns to direct specific data streams to their intended destinations.
Its capabilities cover wide-ranging data collection via HTTP, TCP, UDP, and local file tailing, as well as the processing of system logs following RFC standards. The system includes internal monitoring for throughput and buffer usage, as well as mechanisms for event buffering and compression to prevent data loss during traffic spikes or output failures.
Pipelines are defined using a structured configuration format supporting YAML or a domain-specific language.