influxdata/telegraf
Telegraf
Telegraf is a modular, cross-platform telemetry pipeline designed to collect, process, and route metrics from diverse infrastructure, applications, and hardware. It functions as a server-side middleware that normalizes heterogeneous data into a unified format, enabling consistent monitoring across complex environments. By utilizing a plugin-driven architecture, the agent manages the entire lifecycle of telemetry data from initial ingestion to final transmission.
The project distinguishes itself through a declarative, configuration-driven execution model that allows users to define complex data flow topologies. It supports highly granular control over data processing, including statistical aggregation, transformation, and field mapping, which can be extended through custom scripts or external binaries. To ensure reliability, the agent tracks individual data points through the pipeline, providing delivery confirmation to downstream storage systems and monitoring platforms.
The capability surface covers a vast array of domains, including containerized environments, industrial IoT protocols, distributed message queues, and network performance observability. It includes specialized collectors for cloud services, databases, and system-level hardware metrics, alongside robust security features such as certificate-based authentication and secure credential injection. The agent can be deployed as a persistent background service or orchestrated within containerized clusters, with options to optimize the executable footprint by compiling only the necessary plugins.
Features
- Observability Agents - Collects system, application, and infrastructure performance data to provide visibility into operational health and resource utilization.
- Data Ingestion - Bridges disparate data sources and monitoring backends by normalizing incoming telemetry into structured formats for analysis.
- Telemetry Collectors - Gathers, processes, and aggregates telemetry data from diverse systems, services, and hardware for transmission to monitoring platforms.
- System Performance Monitors - Monitors hardware and operating system metrics including processor usage, disk activity, and network interface status.
- Telemetry Data Pipelines - Ingests metrics from various sources, applies transformations, and routes them to multiple configured output destinations.
- Metric Routers - Routes collected and processed information to various databases, message queues, or monitoring platforms based on output configurations.
- Plugin-Based Architectures - Extends functionality by loading independent input, processor, and output modules that communicate through a shared internal data structure.
- Logging and Telemetry - Gathers performance and operational metrics from servers, databases, and cloud services into a unified monitoring pipeline.
- Metric Data Ingestion - Ingests data from various sources and transmits it to specified destinations by defining input and output plugins.
- Application Metrics Collection - Provides modular interfaces for collecting telemetry from diverse application runtimes and middleware.
- Data Transformation - Applies mathematical or logical operations to incoming data points to normalize, filter, or enrich metrics.
- Configuration-Driven Orchestrators - Defines agent behavior and data flow topology through a centralized, declarative configuration file parsed at runtime.
- Container Monitoring - Collects resource usage, health status, and logs from container orchestration platforms and their running workloads.
- Delivery Confirmations - Tracks metrics through the pipeline and confirms successful transmission to ensure reliable end-to-end data delivery.
- Data Processing Pipelines - Routes telemetry through a sequential flow of collection, transformation, and transmission stages to normalize and deliver metrics.
- Industrial IoT Ingestion - Polls and captures real-time sensor data and operational metrics from industrial automation devices and communication middleware.
- Metrics Collection - Retrieves statistical data from cloud monitoring services and formats them as structured metrics for downstream processing.
- Monitoring and Observability - Monitors traffic flows, interface statistics, and protocol health across network devices and infrastructure components.
- Metric Streaming - Ingests and decodes real-time HTTP metric streams from cloud-based monitoring services.
- AI Tools - Metric collection agent for system monitoring.
- Data Collection Agents - Plugin-driven agent for collecting and reporting system metrics.
- Databases & Data - Metrics collection agent.
- Logging And Aggregation - Plugin-driven agent for metrics collection.
- Metrics Collection - Plugin-driven agent for collecting and processing metrics.
- Data Collection Agents - Plugin-driven server agent for reporting metrics.
- Monitoring and Performance - Provides metrics collection for PostgreSQL databases.
- Batch Aggregators - Combines individual data points into summary statistics over time windows to reduce data volume.
- Statistical Aggregators - Computes aggregate values including counts, differences, minima, maxima, and means for incoming data streams.
- Data Normalization - Converts diverse data formats from heterogeneous sources into a unified internal representation for consistent processing and storage.
- Message Broker Consumers - Consumes and processes data streams from message brokers to ensure reliable delivery and transformation of telemetry.
- Credential Security - Fetches sensitive authentication information like tokens from external secret management systems to protect access.
- Metric and Performance Monitors - Retrieves performance statistics and resource measurements from cloud monitoring services.
- Database Performance Metrics - Gathers operational statistics and performance data from database instances by querying internal system views.
- Resource Metrics - Gathers resource usage, network traffic, and block I/O statistics from containers running within orchestration tasks.
- Cloud - Retrieves performance and health data for cloud resources by querying provider monitoring APIs.
- Service Metrics Monitoring - Collects resource statistics and health metrics from managed cloud service monitoring APIs.
- System Metrics - Gathers granular performance data and usage statistics for individual processor cores and system totals.
- Consumer Lag Monitoring - Collects consumer group status, partition offsets, and lag metrics from monitoring services via HTTP.
- Percentile Calculation - Aggregates numeric fields into specific statistical quantiles to identify distribution percentiles and outliers.
- Container Metric Collectors - Retrieves resource usage statistics and logs from container engines and orchestration platforms by querying native interfaces.
- Kubernetes Application Deployments - Orchestrates across clusters using standard packaging tools to run as single instances or managed operators.
- Access Authentication - Secures connections to clusters using service accounts or tokens to ensure authorized data collection.
- Client Certificate Authentication - Manages client-side certificates to establish secure connections and authenticate with remote servers.
- Cloud Authentication Strategies - Supports multiple authentication methods including roles and environment variables for cloud provider interfaces.
- Machine Identity Authentication - Uses managed identities to securely connect to database instances without requiring hardcoded passwords.
- Latency Histograms - Groups field values into defined ranges to create frequency distributions for analyzing data spread.
- Agent Performance Monitoring - Tracks internal request counts, processing latency, and error rates to diagnose connectivity or ingestion issues.
- Metric Collection - Executes external scripts or commands at regular intervals and parses the resulting output into structured data.
- Service Uptime Monitors - Tests connections to remote servers and reports response times and connection success status as metrics.
- Log Forwarders - Retrieves log output from running containers using engine interfaces and forwards the data for processing.
- Cluster Monitoring Systems - Gathers system, container, and application performance data from distributed cloud operating system clusters.
- Health Monitoring Endpoints - Collects availability metrics from web server upstream modules by tracking successful and failed check attempts.
- Telemetry Collection and Aggregation - Converts incoming dial-out telemetry data from network devices into structured metrics.
- System Monitoring - Queries remote APIs of monitoring systems to retrieve status information for hosts, services, and internal components.
- Scriptable Aggregators - Executes user-defined scripts to process and aggregate incoming metrics using sandboxed languages.
- JMX Metric Collectors - Queries REST endpoints to retrieve telemetry data from Java MBeans.
- Packet Processing Metrics - Collects device statistics and link status from packet processing application telemetry sockets.
- Metric Template Mappers - Transforms dot-delimited strings into structured metric formats using configurable patterns to ensure consistent data representation.
- Message Brokers - Gathers performance and status data from message queues, topics, and subscribers by querying console APIs.
- Message Queue Metric Consumption - Reads metrics or message payloads from message brokers and distributed queue services using standard protocols.
- Distinct Value Counters - Calculates the frequency of distinct values within specified fields over defined time intervals.
- Plugin Execution Engines - Runs standalone programs as plugins to extend data collection or processing capabilities beyond the built-in feature set.
- Environment Variable Configurations - Substitutes environment variables into configuration files to securely manage credentials without hardcoding sensitive information.
- Kubernetes Cluster Management - Gathers operational data and resource status from Kubernetes cluster objects including nodes, deployments, and persistent volumes.
- External Process Plugins - Executes standalone binaries or scripts as plugins to perform specialized tasks by exchanging data via standard input and output streams.
- NATS Message Ingestion - Subscribes to subjects on messaging servers to ingest data, supporting parallel processing and multiple authentication methods.
- Load Balancing Metrics - Gathers performance statistics and status information from load balancer instances to inform traffic distribution.
- Mail Queue Monitoring - Collects performance data from local mail server instances including message counts and queue age.
- Message Stream Consumer Groups - Reads data from message topics using consumer groups to enable parallel processing across multiple instances.
- Network Flow Analyzers - Decodes network traffic flows to extract and structure flow metrics from network devices.
- Cloud Messaging Metric Ingestion - Retrieves messages from cloud-based pub-sub services and transforms them into structured metrics.
- Hypervisor Metrics - Collects performance and state statistics from hypervisors without requiring agent installation on guest machines.
- Derivative Calculators - Computes the rate of change for metric fields over time to normalize derivative calculations.
- Firewalls - Gathers packet and byte counters from firewall rules to monitor network traffic and security state.
- TLS Transfer Security - Encrypts incoming traffic using certificates and enforces client authentication through trusted authorities.
- Event Hub Metric Ingestion - Reads data streams from cloud messaging services and IoT hubs to ingest telemetry or event logs.
- Hardware Performance Monitoring - Gathers low-level processor events from system subsystems to monitor hardware utilization and identify bottlenecks.
- Processor Cores - Tracks cache occupancy, memory bandwidth, and instruction execution metrics for processor cores.
- Health Checks - Retrieves the status of registered health checks from service mesh APIs and reports them as telemetry data.
- Connection Health Monitors - Tracks successful and failed query attempts for each configured database connection to alert on failures.
- Internet Performance Monitors - Collects network metrics including speed, latency, jitter, and packet loss by performing tests against external services.
- Metric Field Mergers - Combines multiple metrics sharing the same series and timestamp into single records containing the union of all fields.
- Cache Performance Metrics - Gathers operational statistics and request processing data from protocol routers to monitor cache scaling.
- Infrastructure Metrics - Gathers performance and status data from compute, storage, and networking services within OpenStack cloud environments.
- NVIDIA GPUs - Gathers hardware performance data including memory usage, temperature, and utilization by interfacing with system utilities.
- Server Metrics - Queries node, namespace, set, and histogram statistics from Aerospike servers and normalizes the data.
- Directory Service Metrics - Gathers operational statistics and performance counters from directory backends to track server health.
- Network Monitoring Widgets - Collects throughput, packet counts, and error statistics for network interfaces to track bandwidth usage.
- Process Manager Metrics - Collects operational statistics from process managers via status pages and local sockets.
- GPU Performance Monitoring - Collects real-time hardware metrics including temperature, power consumption, and clock speeds from graphics processing units.
- Query Performance Monitoring - Measures response times and tracks result codes for queries to provide visibility into network service health.
- Storage Monitoring - Provides specialized collectors for querying performance and health data from storage arrays via REST APIs.
- Request Body Parsers - Accepts incoming data via HTTP APIs and parses line protocol bodies to convert requests into internal metric formats.
