Awesome GitHub RepositoriesLogging And Aggregation

Elasticsearch is a distributed search engine and document store designed for the high-performance indexing and retrieval of massive volumes of unstructured data. It functions as a centralized analytics platform, providing a schema-flexible architecture that organizes information into searchable indices while maintaining global cluster state through a distributed consensus mechanism. The platform distinguishes itself through its integrated approach to observability, security, and advanced analytics. It combines full-text, vector, and hybrid search capabilities with machine learning-driven insi

Loki is a horizontally scalable, highly available log aggregation engine designed to store and query massive volumes of unstructured log data. It functions as a distributed observability platform that correlates logs, metrics, and traces to provide comprehensive visibility into the health and performance of complex infrastructure. The system distinguishes itself through a distributed query execution model that processes large datasets in parallel across cluster nodes. It utilizes label-based stream indexing and a distributed index to map log data to specific chunks, enabling rapid retrieval w

Vector is a high-performance observability data pipeline designed to collect, transform, and route logs, metrics, and traces across distributed infrastructure. It functions as a modular engine that decouples data ingestion from processing and transmission, utilizing a component-based architecture to connect diverse sources to multiple destinations. The project distinguishes itself through a focus on reliability and flow control. It implements backpressure-aware data movement to prevent data loss during traffic spikes and utilizes disk-backed event buffering to ensure durability during network

Telegraf is a modular, cross-platform telemetry pipeline designed to collect, process, and route metrics from diverse infrastructure, applications, and hardware. It functions as a server-side middleware that normalizes heterogeneous data into a unified format, enabling consistent monitoring across complex environments. By utilizing a plugin-driven architecture, the agent manages the entire lifecycle of telemetry data from initial ingestion to final transmission. The project distinguishes itself through a declarative, configuration-driven execution model that allows users to define complex dat

Fluentd is a unified logging layer and distributed event router that collects, parses, and routes log data from diverse sources to various storage backends. It functions as a log forwarding agent and pipeline orchestrator, transforming raw unstructured log strings into formatted objects using structured log parsing. The project utilizes a plugin-based pipeline architecture to route data through independent input, filter, and output stages. It differentiates itself through tag-based event routing, which uses regular expression patterns to direct specific data streams to their intended destinat

Beats is a collection of lightweight, modular agents designed to gather, process, and forward operational telemetry from distributed infrastructure to centralized storage and analysis platforms. These agents function as a distributed data transport layer, decoupling the collection of logs, metrics, and network events from their final delivery destination. By maintaining local state and managing data flow, the system ensures reliable transmission of information across heterogeneous environments. The project distinguishes itself through a modular pipeline architecture that allows for the assemb

Quickwit is a cloud-native, distributed search engine designed for observability data such as logs, traces, and metrics. It functions as an observability backend that decouples compute from storage by persisting indices directly in S3-compatible cloud object stores. The system is distinguished by its compatibility with the Elasticsearch REST API, allowing it to integrate with existing clients and log shippers without reconfiguration. It also serves as an OpenTelemetry data indexer, ingesting technical data via the OpenTelemetry Protocol using gRPC and HTTP. The engine utilizes a hybrid of co

Fluent Bit este un log shipper cloud-native și un colector unificat de telemetrie conceput ca un pipeline de date eficient din punct de vedere al resurselor. Acesta ingerează log-uri, metrici și urme (traces) din surse multiple, procesându-le în timp real înainte de a direcționa datele către backend-uri de stocare externe. Proiectul funcționează ca un procesor de flux în timp real și procesor de log-uri OpenTelemetry, capabil să transforme și să filtreze datele folosind SQL și logică condițională. Acționează, de asemenea, ca un agent de tracing distribuit care poate eșantiona urmele pentru a reduce volumul de date, păstrând în același timp căile complete ale cererilor. Sistemul oferă livrarea fiabilă a datelor prin buffering bazat pe sistemul de fișiere și logică de reîncercare cu stare pentru a preveni pierderea datelor în timpul întreruperilor. Arhitectura sa modulară suportă plugin-uri de input și output interschimbabile, rutare bazată pe metadate și capacitatea de a extinde funcționalitatea prin biblioteci partajate. Software-ul poate fi implementat ca un container pe diferite arhitecturi CPU și sisteme de operare.

Heapster enables Container Cluster Monitoring and Performance Analysis for Kubernetes (versions v1.0.6 and higher), and platforms which include it.

Collect logs for docker containers

A lightweight, cloud-native data transfer agent and aggregator