Awesome GitHub RepositoriesBatch Aggregators

StatsD is a metrics aggregator and UDP collection server that collects system counters and timers. It functions as a time-series data forwarder, receiving high-frequency metric updates via a lightweight line protocol and summarizing them before flushing the data to a backend. The project features a pluggable metrics backend framework, allowing aggregated statistics to be routed to various third-party monitoring services or time-series databases such as Graphite. It supports horizontal scaling and high availability through a proxy ring distribution system that forwards incoming packets across

Telegraf is a modular, cross-platform telemetry pipeline designed to collect, process, and route metrics from diverse infrastructure, applications, and hardware. It functions as a server-side middleware that normalizes heterogeneous data into a unified format, enabling consistent monitoring across complex environments. By utilizing a plugin-driven architecture, the agent manages the entire lifecycle of telemetry data from initial ingestion to final transmission. The project distinguishes itself through a declarative, configuration-driven execution model that allows users to define complex dat

Data-Juicer is an open-source framework for cleaning, filtering, deduplicating, and transforming multimodal datasets to prepare them for training large language and vision models. It functions as a distributed data pipeline engine that runs processing jobs across Ray clusters, handling billions of samples with automatic operator fusion and adaptive parallelism. The framework provides a library of operators that leverage large language models for semantic extraction, filtering, and data synthesis within processing pipelines. The project distinguishes itself through a YAML-based data recipe sys

Anomalib is a PyTorch-based library for visual anomaly detection, offering a modular framework, a comprehensive model zoo, and a benchmarking suite designed for industrial defect detection. It provides a wide range of algorithms—including generative, discriminative, teacher-student, and vision-language approaches—that support unsupervised, few-shot, and zero-shot settings. The library enables deployment through model export to ONNX and OpenVINO for edge devices, and includes a no-code web application for training and inference. It also features a command-line interface for orchestrating multi

MMF is a modular framework for building, training, and evaluating vision-and-language models. It provides a configuration-driven experiment system where model, dataset, and training parameters are defined through composable YAML files, alongside a curated model zoo of pretrained checkpoints for state-of-the-art multimodal architectures. The framework includes a multimodal dataset loader that downloads, processes, and batches vision-and-language data, and a vision-language model trainer supporting distributed training, mixed precision, and checkpoint-based resumption. The framework distinguish

Flashlight ist eine C++-Bibliothek für maschinelles Lernen und ein Deep-Learning-Framework zur Erstellung und zum Training neuronaler Netze. Es fungiert als Tensor-Manipulationsbibliothek und Engine für automatische Differenzierung, die Operationen verfolgt, um Gradienten via Backpropagation für die Modelloptimierung zu berechnen. Das Projekt zeichnet sich durch seine Rolle als Framework für verteiltes Training aus, das All-Reduce-Gradientensynchronisation und verteilte Umgebungen nutzt, um Machine-Learning-Workloads über mehrere Nodes und Geräte hinweg zu skalieren. Es verfügt über eine Backend-agnostische Speicherschnittstelle und RAII-basiertes Management, um Tensor-Operationen von der physischen Hardware zu entkoppeln. Das Framework deckt ein breites Funktionsspektrum ab, einschließlich der Konstruktion neuronaler Netzwerkarchitekturen mit konvolutiven, linearen und rekurrenten Schichten. Es bietet umfangreiche Utilities für Tensor-Algebra, Dataset-Management und Batching, versionierte Binärserialisierung für Modellzustände sowie Überwachungswerkzeuge zur Verfolgung von Trainingsmetriken und Speicherauslastung.