Awesome GitHub RepositoriesDistributed Task Workers

Airflow is a workflow orchestration platform for authoring, scheduling, and monitoring complex data pipelines as code using Python. It employs a DAG-based task scheduler to manage execution timing and dependencies via directed acyclic graphs, utilizing a distributed task execution engine to run workloads across a cluster of worker nodes. The platform provides a data pipeline monitor for tracking the health and execution history of programmatic workflows. This includes a web interface for workflow progress visualization and health monitoring to identify and troubleshoot pipeline failures. The

Bull is a Node.js library for managing distributed jobs and message queues using Redis as the primary data store. It functions as a distributed task worker, job scheduler, and priority queue manager designed to handle asynchronous workloads across multiple processes. The project distinguishes itself by providing a persistent communication channel that decouples servers through the exchange of serializable data objects. It ensures distributed system reliability by detecting stalled tasks and recovering from process crashes to ensure every queued job is completed. The system covers a broad ran

QuantAxis is a quantitative trading platform and algorithmic trading framework. It provides a comprehensive local environment for backtesting strategies, managing financial market data, and executing trades across stocks, futures, and options markets. The system distinguishes itself through a distributed task scheduler that spreads asynchronous computations and heavy mathematical workloads across a network of remote agents. It incorporates a multi-account trading interface to standardize the monitoring of positions and the execution of orders across various brokerage accounts. The platform c

rq is a distributed task queue and background worker system for Python that uses a Redis backend to decouple task submission from execution. It functions as a reliable message queue and task scheduler, allowing Python functions or asyncio coroutines to be processed asynchronously across multiple worker processes. The project distinguishes itself through reliable queuing mechanisms that prevent job loss during worker crashes using atomic operations. It provides specialized orchestration capabilities, including the prevention of duplicate jobs, job execution prioritization, and the ability to m

Agenda is a persistent background job scheduler and distributed task runner for Node.js applications. It functions as a cron job manager and task queue that ensures background processes survive application restarts by storing job state and metadata in a database. The system coordinates execution across multiple worker instances using distributed locking mechanisms to prevent duplicate processing. It supports flexible scheduling via cron expressions or specific dates and includes a pluggable storage interface for backends such as MongoDB, PostgreSQL, and Redis. The platform provides controls

Resque is a Ruby library for enqueueing and processing asynchronous tasks using Redis as a data store. It functions as a distributed task processor and queue manager, allowing long-running work to be moved out of the main request cycle. The system executes background jobs in isolated child processes to prevent memory leaks and provides a web-based dashboard for monitoring queue depths, worker activity, and failed job statistics. Capability areas include distributed worker coordination via signals, error handling with job retry mechanisms, and priority-ordered queue management. It also suppor

Cog is a machine learning packaging tool and containerized model wrapper that bundles models and their dependencies into standardized Docker containers. It functions as an environment manager and inference server, ensuring consistent model execution across different hardware systems by resolving GPU drivers, system libraries, and Python dependencies. The project distinguishes itself by automatically generating RESTful HTTP servers and OpenAPI schemas based on defined model input and output types. It manages large model weights as external fixtures to optimize image size and utilizes a slot-ba

OpenCTI is a cyber threat intelligence platform and knowledge base used to store, manage, and analyze technical security data. It functions as a threat intelligence visualization tool and an enterprise security data orchestrator that maps relationships between threat actors, malware, and vulnerabilities. The platform utilizes the STIX and TAXII standards for data representation and exchange, allowing for the sharing and receiving of standardized intelligence bundles. It distinguishes itself by converting complex security information into visual relationship diagrams and geographic maps to ide

Evolver is a self-evolving AI agent framework that uses gene expression programming to autonomously improve agent behaviors through a continuous five-step loop of scanning, selecting, mutating, validating, and solidifying. It functions as an auditable evolution system that records every mutation and selection step, and can translate natural-language problems into executable Python code for automated grading and evaluation. The framework distinguishes itself through a distributed architecture that enables multiple agents to collaborate and share learned experiences across a network. It operate

BullMQ is a Redis-backed message queue library and background processor designed for distributed task queueing. It functions as a distributed queue manager and task scheduler, utilizing Redis to manage asynchronous job processing and persistence. The system distinguishes itself through its role as a job workflow orchestrator, enabling the definition of complex parent-child job dependencies and hierarchies for multi-step workflows. It provides sandboxed process execution to isolate heavy workloads and prevent event loop blocking, alongside distributed rate limiting to protect downstream servic

Machinery is a distributed task queue and asynchronous workflow engine. It provides a system for processing heavy workloads outside the main request flow using a network of distributed background workers and a message-based job orchestrator. The project manages complex task lifecycles through sequential chaining, where results are passed between tasks, and parallel coordination, which can trigger callback tasks upon the completion of a group. It supports periodic workflow scheduling for recurring jobs and delayed execution via specific timestamps. The system includes capabilities for result

Hyperopt is a Python library for hyperparameter optimization designed to minimize scalar-valued objective functions. It operates as a stochastic search space engine that finds optimal input parameters by searching through real-valued, discrete, and conditional spaces. The framework distinguishes itself through its support for complex search space configurations, allowing for conditional parameter hierarchies where specific hyperparameters are sampled only if their parent parameters meet certain criteria. It is built as an asynchronous optimization framework, decoupling the generation of searc

This project is a collection of learning resources and instructional guides for implementing asynchronous messaging patterns using RabbitMQ. It provides a series of tutorials and runnable code examples focused on the Advanced Message Queuing Protocol to help users decouple services via a message broker. The resources cover practical implementation patterns including request-reply, pub-sub, and stream processing. These guides demonstrate how to use official client libraries to balance worker loads, route messages across multiple consumers in a distributed system, and deploy high availability b

Osmedeus is a security workflow orchestration engine that coordinates AI agents, shell commands, and scanning tools through declarative YAML pipelines. It functions as a distributed security scanner, a declarative workflow automator, and an AI agent framework for security, enabling automated multi-step security analysis with conditional branching, parallel execution, and distributed workers. The engine distinguishes itself through a hybrid runner model that executes workflow steps on the local host, inside Docker containers, or over SSH to remote machines, selected per step or module. It supp

Flashlight ist eine eigenständige C++-Bibliothek für maschinelles Lernen und Tensor-Berechnungen, die zum Erstellen und Trainieren neuronaler Netze verwendet wird. Sie fungiert als umfassendes Framework für neuronale Netze und Engine für automatische Differenzierung und bietet Werkzeuge zur Konstruktion von Berechnungsgraphen und zur Berechnung von Gradienten via Backpropagation. Das Projekt dient als Framework für verteiltes Training und nutzt All-Reduce-Operationen zur Synchronisation von Gradienten und Parametern über mehrere Rechenknoten und Geräte hinweg. Es zeichnet sich durch eine tiefe Integration von leistungsstarker Tensor-Manipulation, nativer Interoperabilität mit Gerätespeichern und einem System zur Synchronisation von Gewichten über verteilte Worker aus, um das Training großskaliger Modelle zu beschleunigen. Das Framework deckt eine breite Palette an Deep-Learning-Funktionen ab, einschließlich modularer Schichtkomposition für den Entwurf komplexer Architekturen wie Residual-Blöcke und rekurrente Zellen. Es bietet umfangreiche Datenmanagement-Utilities für Ingestion und Prefetching sowie Serialisierungssysteme zur Persistierung von Modellzuständen. Zusätzlich enthält es eine Suite an Überwachungs- und Observability-Tools zur Verfolgung von Trainingsmetriken und zur Messung von Sequenzfehlern. Die Bibliothek ist in C++ implementiert.

Dramatiq is a distributed task queue and workload manager used to offload function execution to background workers. It functions as an asynchronous task orchestrator that enables the distribution of computational tasks across a cluster using a pluggable transport layer supporting RabbitMQ and Redis. The framework provides specialized tools for complex task orchestration, including the ability to link background jobs into sequences, pipelines, and barriers. It further manages distributed concurrency through the use of shared mutexes, rate limiters, and exponential backoff retries to prevent re

PGMQ ist ein leichtgewichtiges Message-Queuing-System, das als PostgreSQL-Erweiterung implementiert ist, um asynchrone Aufgaben zu verwalten. Es fungiert als datenbankgestützter Message-Broker, der PostgreSQL für persistente Speicherung, atomare Operationen und benachrichtigungsgesteuerte Zustellung nutzt. Das System bietet ein SQS-kompatibles Queue-Modell mit Sichtbarkeits-Timeouts und verzögerter Zustellung. Es unterstützt eine strikte First-in-First-out-Reihenfolge durch Gruppen-Keys und Batch-Abruf, um die sequentielle Verarbeitung verwandter Aufgaben sicherzustellen. Das Projekt deckt den gesamten Lebenszyklus einer Nachricht ab, einschließlich Produktion, Konsum über atomare Pop-Operationen und administrative Verwaltung wie Queue-Bereinigung und Lebenszykluskontrolle. Es enthält Routing-Funktionen durch themenbasierte Muster und Zuverlässigkeitsmerkmale wie Dead-Letter-Queueing, Nachrichtenarchivierung und Wiederholungslogik. Überwachungstools werden bereitgestellt, um operative Metriken wie Queue-Länge und Durchsatz zu verfolgen.

AFL is a coverage-guided fuzzer and security vulnerability scanner used to identify software bugs and memory corruption by feeding programs mutated data. It functions as a binary instrumentation tool and a test case minimizer to locate crashes and isolate the smallest set of bytes causing a fault. The project distinguishes itself through its ability to operate as a parallel fuzzing orchestrator, distributing workloads across multiple CPU cores or networked machines. It utilizes dictionary-based mutation for complex file formats and performs input sensitivity analysis to identify critical sect

stairspeedtest-reborn is an asynchronous network testing engine and performance analyzer designed to benchmark proxy connectivity and speed. It functions as a diagnostic platform for measuring latency and throughput across multiple Shadowsocks and V2Ray proxy nodes, providing a web-based interface for configuration and visualization. The system optimizes proxy routing by evaluating the stability and efficiency of various network paths through automated batch testing. It utilizes a concurrent worker system that executes isolated subprocesses to ensure individual node tests do not interfere wit