Awesome GitHub RepositoriesLog Object Storage

Vector is a high-performance observability data pipeline designed to collect, transform, and route logs, metrics, and traces across distributed infrastructure. It functions as a modular engine that decouples data ingestion from processing and transmission, utilizing a component-based architecture to connect diverse sources to multiple destinations. The project distinguishes itself through a focus on reliability and flow control. It implements backpressure-aware data movement to prevent data loss during traffic spikes and utilizes disk-backed event buffering to ensure durability during network

VictoriaMetrics is a high-performance, scalable time series database and observability platform designed for long-term storage and analysis of metric, log, and trace data. It functions as a unified backend for monitoring ecosystems, offering full compatibility with industry-standard protocols and query languages. The system is built to handle massive data volumes through a distributed architecture that supports horizontal scaling and efficient data lifecycle management. The platform distinguishes itself through a storage engine that utilizes consistent hashing for data sharding and log-struct

Thanos is a distributed metrics query engine and monitoring scalability suite designed to provide a unified interface for aggregating data from multiple Prometheus servers and clusters. It functions as a high availability monitoring backend that eliminates single points of failure by deduplicating data from replicated instances. The system enables long-term retention by persisting time-series data to cloud-native object storage, allowing for unlimited historical archiving beyond the limits of local disks. It further optimizes this storage through a downsampling and retention manager that comp

Thanos is a CNCF cloud native monitoring tool that provides a highly available and scalable extension to the Prometheus ecosystem. It functions as a global query engine, a long-term storage system, and a metric downsampler. The project enables a unified interface to aggregate and query metrics across multiple distributed clusters from a single view. It maintains historical data beyond local retention limits by persisting time-series metrics in object storage and eliminates data gaps by merging metrics from redundant server pairs. The system includes capabilities for reducing the resolution o

Stalwart is a self-hosted email and collaboration infrastructure that provides an integrated mail server supporting SMTP, IMAP, POP3, and JMAP protocols. It functions as a comprehensive communication hub, combining email hosting with a collaboration server for shared calendars, contacts, and files. The system distinguishes itself through a distributed architecture that uses peer-to-peer cluster coordination to ensure high availability and fault tolerance. It features a built-in security suite that implements an S/MIME and OpenPGP email gateway alongside automated TLS certificate provisioning

PredictionIO is a machine learning server designed for the deployment of predictive models to transform raw data into actionable predictions. It manages the full lifecycle of machine learning operations, from ingesting event data via APIs to hosting production-ready predictive services for real-time inference. The system supports distributed model training by spreading computational workloads across a cluster of nodes to increase processing speed. It enables the implementation of custom prediction engines using programming languages or the application of pre-built model templates for common t

Quickwit is a cloud-native, distributed search engine designed for observability data such as logs, traces, and metrics. It functions as an observability backend that decouples compute from storage by persisting indices directly in S3-compatible cloud object stores. The system is distinguished by its compatibility with the Elasticsearch REST API, allowing it to integrate with existing clients and log shippers without reconfiguration. It also serves as an OpenTelemetry data indexer, ingesting technical data via the OpenTelemetry Protocol using gRPC and HTTP. The engine utilizes a hybrid of co

AutoMQ is a cloud-native streaming platform and Apache Kafka distribution that implements a decoupled compute and storage architecture. It functions as an S3-backed message queue, using object storage as the primary log repository to eliminate dependencies on local disks. The platform utilizes a stateless broker architecture to enable dynamic compute scaling and automated partition balancing. This design allows the system to adjust the number of brokers in seconds and distribute network traffic without requiring manual data migration or partition reassignment. The system provides multi-avail

AutoMQ is a cloud-native streaming platform and Kafka-compatible message broker. It implements the Kafka protocol to provide integration with existing clients and ecosystems while functioning as a message queue that persists data directly to cloud object storage. The system decouples compute from storage, allowing processing power and storage capacity to scale independently. It utilizes a shared-log architecture and object-storage-based persistence to remove dependencies on local disks, which reduces operational costs and eliminates manual disk management. The platform includes mechanisms fo

Accelerate is a PyTorch distributed training library that abstracts the boilerplate required to run models across multiple GPUs, TPUs, and CPUs. It functions as a deep learning model scaler and distributed hardware orchestrator, allowing the same training script to run on different hardware backends without modifying the core logic. The project provides a distributed training command line interface for configuring compute environments and launching jobs across single or multi-node clusters. It includes a mixed precision training framework to implement FP16 and BF16 precision, reducing memory

Databend is a cloud-native data warehouse and OLAP database designed for large-scale analytics. It functions as a SQL-compliant engine and serverless analytics platform that separates compute from storage to allow for independent scaling. The system integrates vector database capabilities, indexing high-dimensional embeddings to enable semantic, hybrid, and full-text searches across massive datasets. It further distinguishes itself through serverless compute management that automatically scales resources based on demand and shuts them down during idle periods. The platform covers a broad set

immudb is a tamperproof database that maintains an immutable record of entries using cryptographic commit logging. It ensures verifiable database integrity by utilizing Merkle trees to generate membership and consistency proofs that detect unauthorized data alterations. The system employs a multi-model storage engine that unifies key-value, document, and relational data structures within a single immutable backend. It provides compatibility with the PostgreSQL wire protocol, allowing it to integrate with standard SQL clients, ORMs, and database tools. The project covers broad capabilities in

OpenWhisk is a serverless cloud platform designed for deploying and executing stateless functions in response to API calls or events. It serves as a complete serverless stack, providing an API gateway for functions, a function-as-a-service runtime manager, and an event-driven workflow engine. The platform distinguishes itself through a polyglot execution model that supports multiple language runtimes and allows for the creation of custom runtimes using Docker containers. It enables complex logic through function orchestration and composition, allowing multiple functions to be chained into seq

GreptimeDB is a distributed, open-source time-series database built for unified observability. It stores and queries metrics, logs, and traces together in a single columnar engine, supporting both SQL and PromQL for analysis. The database is designed as a Kubernetes-native operator with a decoupled compute and storage architecture, enabling horizontal scaling and multi-region deployment. What distinguishes GreptimeDB is its role as a multi-protocol ingestion gateway, accepting data through OpenTelemetry, Prometheus Remote Write, InfluxDB, Loki, Elasticsearch, Kafka, and MQTT protocols without

Cortex is an open-source, horizontally scalable metrics platform that ingests, stores, and queries Prometheus-compatible time-series data with multi-tenant isolation. It accepts metrics via Prometheus remote write and OpenTelemetry, executes PromQL queries against both recent and historical data, and provides a Prometheus-compatible alerting and recording rule engine with an integrated Alertmanager. The system is built as a set of independently scalable microservices that use hash-ring-based sharding, gossip-based cluster membership, and tenant-aware object storage to distribute workloads acro

Devtron ist eine Kubernetes-Management-Plattform und ein CI/CD-Orchestrator, der entwickelt wurde, um Anwendungslebenszyklen und Infrastrukturoperationen über mehrere Cluster hinweg von einem einzigen Interface aus zu vereinheitlichen. Es dient als zentrales Dashboard für die Orchestrierung von Workloads, das Management von Sicherheit und die Bereitstellung von Observability für Kubernetes-Umgebungen. Die Plattform zeichnet sich durch eine No-Code-Workflow-Engine zur Automatisierung von Container-Builds und Software-Delivery-Pipelines aus, neben einem visuellen GitOps-Deployment-Tool für die Verwaltung deklarativer Anwendungen und die Abstimmung von Konfigurations-Drifts. Ihre Funktionsoberfläche erstreckt sich auf das Sicherheitsmanagement durch das Scannen von Container-Images auf Schwachstellen und die Synchronisierung externer Secrets sowie auf Observability über ein Dashboard, das Metriken, Logs und Events für das Debugging korreliert. Das System handhabt zudem Infrastrukturaufgaben wie eventgesteuerte Workload-Skalierung, Ressourcen-Hibernation und rollenbasierte Zugriffskontrolle. Die Plattform kann mittels eines Bootstrap-Installationsprozesses bereitgestellt werden, der alle erforderlichen Komponenten und Drittanbieter-Abhängigkeiten verwaltet.

Grafana Tempo is a high-scale distributed tracing backend and columnar trace database. It serves as an observability data store that persists and queries spans and traces using OpenTelemetry standards, allowing for the analysis of request flows across microservices. The system distinguishes itself by using an object-store based backend with columnar Parquet storage. This architecture enables efficient attribute searching and large-scale data retrieval through dedicated attribute columnization and block-based data partitioning. It includes a specialized TraceQL query engine for filtering trace

Telegram Search ist eine selbstgehostete Plattform zum Exportieren, Indizieren und Archivieren von persönlichen oder Gruppen-Nachrichtenverläufen. Sie fungiert als private Suchmaschine, die verstreute Kommunikationsprotokolle und Medien-Assets in eine durchsuchbare Wissensbibliothek umwandelt, sodass Benutzer durch containerisierte Infrastruktur die volle Kontrolle über ihre Daten behalten. Die Plattform zeichnet sich durch die Nutzung vektorbasierter semantischer Indizierung aus, um eine Fuzzy-Suche über historische Datensätze zu ermöglichen. Sie integriert eine OCR-Pipeline (Optical Character Recognition), um Text aus Bildern und Mediendateien zu extrahieren und sicherzustellen, dass visuelle Inhalte ebenso auffindbar sind wie textbasierte Nachrichten. Benutzer können direkt aus den Suchergebnissen über protokollbasierte Deep-Links zur ursprünglichen Quelle innerhalb der Messaging-Anwendung navigieren. Über die einfache Suche hinaus integriert das System generative KI, um kontextbezogene Zusammenfassungen und Antworten basierend auf gespeicherten Chat-Protokollen bereitzustellen. Diese RAG-Fähigkeit (Retrieval-Augmented Generation) ermöglicht eine intelligente Analyse historischer Threads, während die automatisierte Medienarchivierung große Assets auf externen Speicher auslagert, um eine schlanke lokale Datenbank zu erhalten. Der gesamte Stack wird über containerisierte Konfigurationen bereitgestellt, um die Verwaltung von Interface, Datenbank und Speicherdiensten zu vereinfachen.

Helix DB is a distributed graph database and knowledge graph platform that persists nodes and edges on object storage for durable and unlimited scaling. It operates as an ACID-compliant system, ensuring data consistency through serializable snapshot isolation during concurrent operations. The project distinguishes itself by combining a vector search engine and a property graph, utilizing hybrid vector and full-text search to locate entry points for graph traversals. It enables dynamic graph querying through a domain-specific language, allowing complex logic and recursive queries to be execute

minio-go is a client library and software development kit for interacting with S3-compatible object storage. It provides a programmatic interface for Go applications to manage buckets and objects using the S3 protocol. The library enables the execution of complex storage operations, including multi-part uploads for large datasets, data synchronization between filesystems, and the management of bucket lifecycle and replication policies. It also supports advanced data retrieval through object searching and SQL-based querying of stored data. The toolkit covers a broad range of administrative an