Awesome GitHub RepositoriesLog Ingestion

Nanoclaw is an LLM agent orchestrator and multi-platform chat gateway designed to deploy and manage isolated AI agents. It provides a containerized runtime that executes agents within sandboxed Linux containers, ensuring filesystem and state isolation through dedicated workspaces and host bind-mounts. The project distinguishes itself through a unified routing pipeline that connects agents to diverse messaging platforms, including WhatsApp, Discord, Slack, Telegram, Signal, and iMessage. It integrates the Model Context Protocol to extend agent capabilities via managed external data and functio

SigNoz is a full-stack observability platform designed to collect, store, and visualize metrics, logs, and distributed traces in a unified environment. It leverages OpenTelemetry-based data collection to ingest telemetry from diverse sources using vendor-neutral protocols, ensuring interoperability across complex microservices architectures. The platform utilizes a high-performance columnar storage engine to enable rapid aggregation and filtering, providing a centralized backend for monitoring application health and performance. What distinguishes the platform is its focus on automated instru

Logrus is a structured logging library for Go that produces machine-readable output using key-value pairs and JSON formatting. It serves as a pluggable logging framework providing a thread-safe event logger with configurable mutex locking to manage concurrent writes across multiple goroutines. The project distinguishes itself through a pluggable hook system that routes log entries to external services or custom destinations. It also features a contextual logger capable of attaching persistent metadata and request-scoped fields to entries to improve traceability. The framework covers broad ob

Winston is a versatile logging library for Node.js designed to record system events and metadata. It functions as a multi-transport log manager that routes data to various destinations and a structured log formatter that transforms entries into JSON or plain text. The project is distinguished by its pluggable transport architecture, which decouples the logging interface from delivery mechanisms. This allows for the creation of custom transport extensions and the use of hierarchical logger instances to inherit configurations while attaching persistent metadata to downstream messages. The libr

Vector is a high-performance observability data pipeline designed to collect, transform, and route logs, metrics, and traces across distributed infrastructure. It functions as a modular engine that decouples data ingestion from processing and transmission, utilizing a component-based architecture to connect diverse sources to multiple destinations. The project distinguishes itself through a focus on reliability and flow control. It implements backpressure-aware data movement to prevent data loss during traffic spikes and utilizes disk-backed event buffering to ensure durability during network

This project is a containerized orchestration layer for the Elastic Stack, providing a pre-configured set of Docker Compose files to deploy Elasticsearch, Logstash, and Kibana as a unified data analysis stack. It functions as a centralized log management system for ingesting, indexing, and searching log data using a cluster of interconnected services. The deployment pattern includes an Elasticsearch cluster manager that enables scaling data nodes through replica scaling and internal discovery. It provides a web-based administration interface for monitoring cluster health and status. The syst

VictoriaMetrics is a high-performance, scalable time series database and observability platform designed for long-term storage and analysis of metric, log, and trace data. It functions as a unified backend for monitoring ecosystems, offering full compatibility with industry-standard protocols and query languages. The system is built to handle massive data volumes through a distributed architecture that supports horizontal scaling and efficient data lifecycle management. The platform distinguishes itself through a storage engine that utilizes consistent hashing for data sharding and log-struct

Quarkus is a Kubernetes-native Java framework designed for building high-performance, memory-efficient applications. It utilizes ahead-of-time native compilation to transform Java code into standalone, optimized binaries that eliminate the need for a virtual machine, enabling rapid startup and reduced memory consumption. By performing code augmentation during the build phase, it shifts heavy processing tasks away from runtime, ensuring that applications are optimized for cloud-native environments. The framework distinguishes itself through a unified approach to reactive and imperative program

Pinpoint is a distributed application performance management tool designed to trace requests and monitor metrics across large-scale distributed architectures. It functions as a request tracer, topology mapper, and JVM application monitor, providing a backend capable of collecting and visualizing trace data from OpenTelemetry compatible sources. The system distinguishes itself through a combination of bytecode-based instrumentation via a Java agent and topology-based visualization that renders live maps of service interconnections. It captures execution flow across asynchronous boundaries, suc

Fluentd is a unified logging layer and distributed event router that collects, parses, and routes log data from diverse sources to various storage backends. It functions as a log forwarding agent and pipeline orchestrator, transforming raw unstructured log strings into formatted objects using structured log parsing. The project utilizes a plugin-based pipeline architecture to route data through independent input, filter, and output stages. It differentiates itself through tag-based event routing, which uses regular expression patterns to direct specific data streams to their intended destinat

CocoaLumberjack is a logging framework designed for Apple platforms, providing a system for capturing, filtering, and managing diagnostic messages within mobile and desktop applications. It serves as a tool for monitoring software health and troubleshooting runtime issues by recording application events across various severity levels. The framework utilizes a modular architecture that allows developers to define custom log levels and route diagnostic output to multiple concurrent destinations, including the system console, local files, and remote servers. It incorporates an asynchronous pipel

Nightingale is a Prometheus-compatible monitoring and alerting platform designed to centralize telemetry management across multiple time-series databases. It functions as a multi-source alerting engine and metric data pipeline that ingests telemetry via remote write protocols and triggers alarms based on data from sources such as Prometheus, Elasticsearch, Loki, and ClickHouse. The system is distinguished by its automated alert healing system, which executes predefined scripts and RPC-based corrective actions when monitoring thresholds are breached. It supports distributed alert processing, a

Beats is a collection of lightweight, modular agents designed to gather, process, and forward operational telemetry from distributed infrastructure to centralized storage and analysis platforms. These agents function as a distributed data transport layer, decoupling the collection of logs, metrics, and network events from their final delivery destination. By maintaining local state and managing data flow, the system ensures reliable transmission of information across heterogeneous environments. The project distinguishes itself through a modular pipeline architecture that allows for the assemb

CrowdSec is a collaborative, distributed security engine designed for threat detection and infrastructure protection. It functions as an intrusion detection system that parses logs and network traffic to identify malicious patterns, utilizing a bucket-based threshold detection model to aggregate events and trigger alerts. The platform is built on a modular architecture that includes a centralized local API server for managing security signals and a relational database for persistent storage of remediation decisions. What distinguishes the project is its decoupled enforcement model, which offl

DataHub is a metadata management platform designed to unify technical, operational, and business context across diverse data ecosystems. By utilizing a graph-based metadata model and an event-driven ingestion architecture, it creates a centralized source of truth that maps complex data relationships, lineage, and ownership. This foundational framework enables organizations to maintain a synchronized view of their data landscape, supporting both human-led discovery and automated data operations. The platform distinguishes itself through its focus on grounding artificial intelligence and autono

This project is a standardized specification for logging APIs in PHP. It defines a common logging interface and a set of severity levels to ensure ecosystem-wide compatibility between different logging implementations. The project focuses on PHP library interoperability by providing a vendor-neutral interface. This allows various software packages to record application events consistently without being tied to a specific logging vendor, adhering to the PSR-3 implementation standard. The specification covers architectural primitives for logging abstractions, including implementation-agnostic

Rerun is a multimodal data visualizer and robotics data logger designed for rendering synchronized streams of 3D spatial data, images, and time-series metrics. It functions as a tool for capturing high-frequency sensor data and AI outputs into a queryable columnar format, providing a dedicated interface for viewing MCAP recording files and analyzing physical environments. The project distinguishes itself as a machine learning dataset streamer, capable of feeding logged recordings directly into GPU buffers and PyTorch training pipelines without intermediate exports. It supports a high-performa

Sigma is a suite of tools for defining generic log signatures and translating them for multiple backends. It provides a structured way to define malicious behavior and detection logic independently of any specific backend technology, acting as a translation engine that maps generic event fields and correlation logic to the proprietary query languages of security data lakes and SIEM platforms. The project features a plugin-based multi-backend query generator that exports security detections into various database and log management formats. It also includes a threat framework mapping tool that

WebDriverIO is a Node.js test automation framework used for automating functional tests across web browsers and mobile applications. It acts as a WebDriver protocol client that manages remote browser sessions and executes commands against WebDriver and Appium servers to perform end-to-end testing. The framework is distinguished by its ability to control both native and hybrid mobile applications and its support for running automated suites across local machines, remote grids, and cloud device providers. It includes specialized capabilities for coordinating multi-browser interactions and estab

Boto3 is the AWS SDK for Python, providing a programmatic interface for managing and automating AWS cloud infrastructure and services. It serves as a cloud management API client and resource manager for provisioning, configuring, and scaling virtual servers, databases, and storage. The library enables the implementation of infrastructure-as-code through declarative templates and scripts, allowing for the deployment of identical resource stacks across multiple accounts and geographic regions. It also provides a framework for coordinating distributed workflows, serverless functions, and contain