elastic/beats
Beats
Beats is a collection of lightweight, modular agents designed to gather, process, and forward operational telemetry from distributed infrastructure to centralized storage and analysis platforms. These agents function as a distributed data transport layer, decoupling the collection of logs, metrics, and network events from their final delivery destination. By maintaining local state and managing data flow, the system ensures reliable transmission of information across heterogeneous environments.
The project distinguishes itself through a modular pipeline architecture that allows for the assembly of specialized agents using shared library building blocks. Each agent is compiled as a statically linked binary, enabling deployment across diverse infrastructure without external runtime dependencies. During the ingestion process, the system automatically enriches raw telemetry with contextual metadata from host systems and cloud environments, while applying backpressure-aware flow control to manage data volume based on destination responsiveness.
The platform covers a broad range of observability tasks, including system performance monitoring, network traffic analysis, and security auditing. It supports the collection of diverse data types such as application logs, Windows event logs, infrastructure metrics, and network packets. Users can filter and parse incoming data streams before forwarding them to centralized storage engines or message queues, ensuring that only relevant information is indexed for long-term analysis.
Features
- Elasticsearch Data Shippers - Ships logs, metrics, and network telemetry from distributed servers to centralized storage and analysis platforms.
- Data Forwarders - Gathers logs, metrics, and network packets from servers for forwarding to centralized storage.
- Log Forwarders - Monitors file paths and transmits log entries to centralized storage for indexing and analysis.
- Observability Collectors - Provides specialized binaries that parse, filter, and forward operational data while maintaining state.
- Telemetry Agents - Deploys modular agents to gather system performance data, audit events, and infrastructure health status.
- Centralized Logging Systems - Aggregates system and application logs from distributed servers into a single repository for analysis.
- Metric and Performance Monitors - Gathers server metrics and service health data to track system performance across diverse environments.
- Awesome List - A community-curated directory that catalogs and links out to other open-source projects, rather than a standalone tool you run yourself.
- Data Collection Agents - Enables the creation of specialized agents to collect and transmit unique data types from infrastructure.
- Backpressure Controllers - Dynamically adjusts ingestion rates based on destination responsiveness to prevent data overflow.
- Network Traffic Analyzers - Captures and decodes application-layer network packets to provide visibility into communication between servers.
- Security Auditing - Tracks user activity, process behavior, and file integrity changes to identify security policy violations.
- Windows Event - Reads and forwards native Windows event logs for centralized troubleshooting and analysis.
- Metric Collection - Gathers operational statistics from operating systems and third-party services to provide visibility into server health.
- Telemetry Collection and Aggregation - Transports and unifies telemetry data from cloud-native services and serverless functions.
- Data Collection Agents - Lightweight shippers for transporting logs and data to storage.
- Logging And Aggregation - Lightweight data shippers for Elasticsearch.
- Metrics Collection - Data shippers for sending system metrics to logging backends.
- Registry-Based State Trackers - Maintains local disk records of read positions to ensure data continuity across service restarts.
- System Activity Auditors - Collects and centralizes audit events from operating system frameworks to track user and process behavior.
- File System Monitors - Tracks and reports changes to critical system files and configuration binaries in real time.
- Metric Data Ingestion - Collects and forwards metrics from monitoring endpoints into centralized storage.
- Availability and Uptime Trackers - Pings network services and endpoints at regular intervals to verify uptime and report status changes.
- Service Uptime Monitors - Probes remote services using standard network protocols to verify reachability and uptime.
- Read State Trackers - Persists read positions to disk to ensure data continuity and seamless resumption after restarts.
- Cloud Metadata Enrichment - Automatically appends contextual metadata from host systems and cloud environments to telemetry streams.
- Cloud Storage Exporters - Ships execution logs and operational data from cloud-native environments to centralized storage.
- Metric Query Languages - Supports executing native query language commands against stored metrics to generate performance insights.
- Static Binaries - Compiles agents into self-contained, statically linked binaries to eliminate external runtime dependencies.
- Modular Pipeline Architectures - Decouples data collection from delivery using a series of independent, configurable processing stages.
- Event Filtering Rules - Applies user-defined criteria to selectively process or discard events before transmission.
- Service Probing - Executes periodic service checks to monitor infrastructure health and network endpoint availability.
- Data Parsing - Translates raw inputs from various sources into structured formats using pre-configured integration modules.
- Network Access Control - Checks service accessibility from various vantage points to confirm that private resources remain protected.
- Shared Library Interfaces - Provides shared library building blocks for assembling specialized data collection agents.
- Request Correlation - Groups individual request and response messages into complete transactions to track application call performance.
