Projects sharing features with Scribe

Logspout is a container log aggregator, forwarder, and routing engine designed to gather logs from Docker containers and route them to centralized external logging systems. It functions as a network-based utility that pipes container output to remote destinations such as syslog or TCP endpoints. The system features a dynamic routing engine that allows the creation and updating of log routing configurations and destination filters in real time via an HTTP interface without restarting the service. It utilizes an adapter-based architecture, enabling the development of custom output modules and t

Security Onion is a security information and event management platform and network security monitoring suite. It functions as an intrusion detection system and a network traffic analysis tool designed to identify malicious activity and network intrusions through signature-based detection and host-based monitoring. The platform integrates a security case management system to organize investigations by tracking detections and grouping related security events. It provides capabilities for full packet capture, network metadata extraction, and the collection and indexing of security logs from dive

lnav is a terminal-based log viewer and analyzer designed for aggregating, filtering, and analyzing multiple log files in a single chronological view. It functions as a console application that can replace the system pager, providing syntax highlighting and document navigation for system or application logs. The project distinguishes itself by mapping unstructured log data to virtual SQLite tables, enabling the use of SQL and PRQL for structured data analysis, aggregations, and relational queries. It further differentiates its capability set through native integration for retrieving and taili

Stern is a Kubernetes command-line debugging tool and log aggregator designed for streaming and filtering logs from multiple pods and containers. It functions as a log tailing utility that consolidates real-time data from distributed resources into a single, color-coded output. The tool uses regular expressions to select pods and filter log content, allowing for the isolation of specific system events. It automatically tracks the lifecycle of resources, using a watch mechanism to attach to new pods that match filter criteria and removing deleted ones from the active view. Capabilities includ

Fluent Bit is a cloud-native log shipper and unified telemetry collector designed as a resource-efficient data pipeline. It ingests logs, metrics, and traces from multiple sources, processing them in real-time before routing the data to external storage backends. The project functions as a real-time stream processor and OpenTelemetry log processor, capable of transforming and filtering data using SQL and conditional logic. It also acts as a distributed tracing agent that can sample traces to reduce data volume while preserving full request paths. The system provides reliable data delivery th

Cerbos is an open-source authorization service that provides a centralized, language-agnostic engine for managing access control. It functions as a policy-as-code platform, allowing teams to define, test, and distribute authorization rules using declarative YAML or JSON configurations. By decoupling access logic from application code, it enables consistent permission enforcement across diverse service stacks. The project distinguishes itself through its ability to translate high-level authorization policies into native database query filters. This capability allows applications to enforce sec

Secor is a service implementing Kafka log persistence

🦎 A multi-protocol edge & service proxy. Seamlessly interface web apps, IoT clients, & microservices to Apache Kafka® via declaratively defined, stateless APIs.

A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

Hadoop Data Integration with various databases, ftp servers, salesforce. Incremental update, dedup, append, merge your data on Hadoop.

Rudder Server is a customer data platform and event routing pipeline designed to collect, transform, and route customer event data from various sources to data warehouses and business tools. It functions as a customer identity resolver, linking identifiers from multiple sources to build a unified identity graph and comprehensive behavioral customer profiles. The system differentiates itself through reverse ETL capabilities, which push processed customer segments and audiences from data warehouses back into operational third-party applications. It also provides a containerized data plane for K

Hadoop log aggregator and dashboard

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

DEPRECATED: Data collection and processing made easy.

A probabilistic data structure service and storage

ingestr is a command-line tool for copying and syncing data between different database engines and third-party platforms without writing custom code. It functions as an ETL pipeline utility that extracts data from diverse sources and loads it into destinations. The tool features a schema-agnostic data loader that maps source fields to destination columns dynamically, removing the need for predefined static table definitions. It also operates as an incremental data synchronizer, updating destination tables by appending new records or merging changes to maintain current datasets. The system pr

Build platforms that flexibly mix SQL, batch, and stream processing paradigms

DocId set compression and set operation library

Netflix's distributed Data Pipeline

Apache Pulsar is a cloud-native distributed pub-sub messaging system designed for high-performance data ingestion. It functions as a geo-replicated data streamer and a multi-tenant event streaming platform, providing a serverless stream processing engine and a tiered storage messaging broker. The system distinguishes itself by separating serving layers from storage layers to allow independent scaling of compute and data retention. It features native geo-replication to synchronize messages across different geographical regions and employs a multi-layered tenant isolation model using authentica

Velociraptor is a digital forensics and incident response platform, endpoint detection and response system, and visibility tool. It provides a query engine and remote forensic collector used to hunt for indicators of compromise and perform triage across a fleet of hosts. The system is distinguished by its specialized query language for interrogating host state and parsing binary files. It features a notebook environment that combines markdown documentation with executable query cells to standardize investigative workflows and enable collaborative reporting. The platform covers a wide range o

Logstash is a JVM-based event processor and extract, transform, load system designed for log data processing pipelines. It functions as a plugin-based data ingestor that collects, transforms, and delivers logs and event data from multiple sources to various destinations. The system utilizes a modular architecture of interchangeable input, filter, and output components to handle real-time data ingestion and enterprise log aggregation. Users can extend the pipeline's functionality by developing custom plugins to support unique data sources or specific transformation logic. The platform covers

This project is a web-based log viewer and management interface designed specifically for Laravel applications. It serves as a centralized dashboard for browsing, searching, and managing system logs without requiring direct access to raw files or SSH. The system functions as a multi-host log aggregator, utilizing a remote proxy to view and index logs from multiple servers in one location. It includes a regular expression-based parser that interprets structured data from non-standard log files, allowing for custom log format definitions and severity level mapping. Broad capabilities include i

toolong is a terminal log viewer and TUI log manager designed for monitoring live log streams and navigating large log files. It functions as a log aggregator and JSONL formatter, capable of merging multiple log files into a single chronological view by automatically detecting timestamps. The application supports the visualization of structured data by pretty printing JSONL files and applying syntax highlighting to common web server log patterns. It handles large-scale data efficiently through virtual-sized scrollable views, allowing users to open compressed logs or files of any size without

Graylog2-server is an open-source centralized log management system and aggregator. It functions as a log analysis platform designed to collect, index, and analyze log data from multiple sources within a centralized searchable index. The system provides capabilities for enterprise log aggregation and infrastructure monitoring. It enables the gathering of logs from various servers and applications to facilitate log data analysis and root cause troubleshooting across a network. The platform utilizes a distributed indexing pipeline and message-queue based ingestion to handle log streams. It inc