Awesome GitHub RepositoriesDistributed Task Workers

Airflow is a workflow orchestration platform for authoring, scheduling, and monitoring complex data pipelines as code using Python. It employs a DAG-based task scheduler to manage execution timing and dependencies via directed acyclic graphs, utilizing a distributed task execution engine to run workloads across a cluster of worker nodes. The platform provides a data pipeline monitor for tracking the health and execution history of programmatic workflows. This includes a web interface for workflow progress visualization and health monitoring to identify and troubleshoot pipeline failures. The

Bull is a Node.js library for managing distributed jobs and message queues using Redis as the primary data store. It functions as a distributed task worker, job scheduler, and priority queue manager designed to handle asynchronous workloads across multiple processes. The project distinguishes itself by providing a persistent communication channel that decouples servers through the exchange of serializable data objects. It ensures distributed system reliability by detecting stalled tasks and recovering from process crashes to ensure every queued job is completed. The system covers a broad ran

QuantAxis is a quantitative trading platform and algorithmic trading framework. It provides a comprehensive local environment for backtesting strategies, managing financial market data, and executing trades across stocks, futures, and options markets. The system distinguishes itself through a distributed task scheduler that spreads asynchronous computations and heavy mathematical workloads across a network of remote agents. It incorporates a multi-account trading interface to standardize the monitoring of positions and the execution of orders across various brokerage accounts. The platform c

rq is a distributed task queue and background worker system for Python that uses a Redis backend to decouple task submission from execution. It functions as a reliable message queue and task scheduler, allowing Python functions or asyncio coroutines to be processed asynchronously across multiple worker processes. The project distinguishes itself through reliable queuing mechanisms that prevent job loss during worker crashes using atomic operations. It provides specialized orchestration capabilities, including the prevention of duplicate jobs, job execution prioritization, and the ability to m

Agenda is a persistent background job scheduler and distributed task runner for Node.js applications. It functions as a cron job manager and task queue that ensures background processes survive application restarts by storing job state and metadata in a database. The system coordinates execution across multiple worker instances using distributed locking mechanisms to prevent duplicate processing. It supports flexible scheduling via cron expressions or specific dates and includes a pluggable storage interface for backends such as MongoDB, PostgreSQL, and Redis. The platform provides controls

Resque is a Ruby library for enqueueing and processing asynchronous tasks using Redis as a data store. It functions as a distributed task processor and queue manager, allowing long-running work to be moved out of the main request cycle. The system executes background jobs in isolated child processes to prevent memory leaks and provides a web-based dashboard for monitoring queue depths, worker activity, and failed job statistics. Capability areas include distributed worker coordination via signals, error handling with job retry mechanisms, and priority-ordered queue management. It also suppor

Cog is a machine learning packaging tool and containerized model wrapper that bundles models and their dependencies into standardized Docker containers. It functions as an environment manager and inference server, ensuring consistent model execution across different hardware systems by resolving GPU drivers, system libraries, and Python dependencies. The project distinguishes itself by automatically generating RESTful HTTP servers and OpenAPI schemas based on defined model input and output types. It manages large model weights as external fixtures to optimize image size and utilizes a slot-ba

OpenCTI is a cyber threat intelligence platform and knowledge base used to store, manage, and analyze technical security data. It functions as a threat intelligence visualization tool and an enterprise security data orchestrator that maps relationships between threat actors, malware, and vulnerabilities. The platform utilizes the STIX and TAXII standards for data representation and exchange, allowing for the sharing and receiving of standardized intelligence bundles. It distinguishes itself by converting complex security information into visual relationship diagrams and geographic maps to ide

Evolver is a self-evolving AI agent framework that uses gene expression programming to autonomously improve agent behaviors through a continuous five-step loop of scanning, selecting, mutating, validating, and solidifying. It functions as an auditable evolution system that records every mutation and selection step, and can translate natural-language problems into executable Python code for automated grading and evaluation. The framework distinguishes itself through a distributed architecture that enables multiple agents to collaborate and share learned experiences across a network. It operate

BullMQ is a Redis-backed message queue library and background processor designed for distributed task queueing. It functions as a distributed queue manager and task scheduler, utilizing Redis to manage asynchronous job processing and persistence. The system distinguishes itself through its role as a job workflow orchestrator, enabling the definition of complex parent-child job dependencies and hierarchies for multi-step workflows. It provides sandboxed process execution to isolate heavy workloads and prevent event loop blocking, alongside distributed rate limiting to protect downstream servic

Machinery is a distributed task queue and asynchronous workflow engine. It provides a system for processing heavy workloads outside the main request flow using a network of distributed background workers and a message-based job orchestrator. The project manages complex task lifecycles through sequential chaining, where results are passed between tasks, and parallel coordination, which can trigger callback tasks upon the completion of a group. It supports periodic workflow scheduling for recurring jobs and delayed execution via specific timestamps. The system includes capabilities for result

Hyperopt is a Python library for hyperparameter optimization designed to minimize scalar-valued objective functions. It operates as a stochastic search space engine that finds optimal input parameters by searching through real-valued, discrete, and conditional spaces. The framework distinguishes itself through its support for complex search space configurations, allowing for conditional parameter hierarchies where specific hyperparameters are sampled only if their parent parameters meet certain criteria. It is built as an asynchronous optimization framework, decoupling the generation of searc

This project is a collection of learning resources and instructional guides for implementing asynchronous messaging patterns using RabbitMQ. It provides a series of tutorials and runnable code examples focused on the Advanced Message Queuing Protocol to help users decouple services via a message broker. The resources cover practical implementation patterns including request-reply, pub-sub, and stream processing. These guides demonstrate how to use official client libraries to balance worker loads, route messages across multiple consumers in a distributed system, and deploy high availability b

Osmedeus is a security workflow orchestration engine that coordinates AI agents, shell commands, and scanning tools through declarative YAML pipelines. It functions as a distributed security scanner, a declarative workflow automator, and an AI agent framework for security, enabling automated multi-step security analysis with conditional branching, parallel execution, and distributed workers. The engine distinguishes itself through a hybrid runner model that executes workflow steps on the local host, inside Docker containers, or over SSH to remote machines, selected per step or module. It supp

Flashlight este o bibliotecă C++ standalone de machine learning și tensori, utilizată pentru construirea și antrenarea rețelelor neuronale. Aceasta funcționează ca un framework cuprinzător de rețele neuronale și motor de diferențiere automată, oferind instrumentele necesare pentru a construi grafuri de calcul și a calcula gradienții prin backpropagation. Proiectul servește drept framework de antrenare distribuită, utilizând operațiuni all-reduce pentru a sincroniza gradienții și parametrii pe mai multe noduri de calcul și dispozitive. Se distinge prin integrarea profundă a manipulării de înaltă performanță a tensorilor, interoperabilitatea nativă a memoriei dispozitivului și un sistem pentru sincronizarea ponderilor între workerii distribuiți pentru a accelera antrenarea modelelor la scară largă. Framework-ul acoperă o gamă largă de capabilități de deep learning, inclusiv compoziția modulară a straturilor pentru proiectarea arhitecturilor complexe precum blocuri reziduale și celule recurente. Oferă utilitare extinse de gestionare a datelor pentru ingestie și prefetching, alături de sisteme de serializare pentru persistența stărilor modelelor. În plus, include o suită de instrumente de monitorizare și observabilitate pentru urmărirea metricilor de antrenare și măsurarea erorilor de secvență. Biblioteca este implementată în C++.

Dramatiq is a distributed task queue and workload manager used to offload function execution to background workers. It functions as an asynchronous task orchestrator that enables the distribution of computational tasks across a cluster using a pluggable transport layer supporting RabbitMQ and Redis. The framework provides specialized tools for complex task orchestration, including the ability to link background jobs into sequences, pipelines, and barriers. It further manages distributed concurrency through the use of shared mutexes, rate limiters, and exponential backoff retries to prevent re

PGMQ este un sistem de coadă de mesaje ușor implementat ca o extensie PostgreSQL pentru gestionarea sarcinilor asincrone. Acesta funcționează ca un broker de mesaje bazat pe baza de date care utilizează PostgreSQL pentru stocare persistentă, operațiuni atomice și livrare bazată pe notificări. Sistemul oferă un model de coadă compatibil cu SQS, cu timeout-uri de vizibilitate și livrare întârziată. Suportă ordonarea strictă first-in-first-out prin chei de grup și regăsirea în loturi pentru a asigura procesarea secvențială a sarcinilor conexe. Proiectul acoperă un ciclu de viață complet al mesajului, inclusiv producția, consumul prin operațiuni pop atomice și gestionarea administrativă, cum ar fi curățarea cozii și controlul ciclului de viață. Include capabilități de rutare prin modele bazate pe subiecte și funcționalități de fiabilitate precum dead letter queueing, arhivarea mesajelor și logica de reîncercare. Sunt furnizate instrumente de monitorizare pentru a urmări metricile operaționale, cum ar fi lungimea cozii și throughput-ul.

AFL is a coverage-guided fuzzer and security vulnerability scanner used to identify software bugs and memory corruption by feeding programs mutated data. It functions as a binary instrumentation tool and a test case minimizer to locate crashes and isolate the smallest set of bytes causing a fault. The project distinguishes itself through its ability to operate as a parallel fuzzing orchestrator, distributing workloads across multiple CPU cores or networked machines. It utilizes dictionary-based mutation for complex file formats and performs input sensitivity analysis to identify critical sect

stairspeedtest-reborn is an asynchronous network testing engine and performance analyzer designed to benchmark proxy connectivity and speed. It functions as a diagnostic platform for measuring latency and throughput across multiple Shadowsocks and V2Ray proxy nodes, providing a web-based interface for configuration and visualization. The system optimizes proxy routing by evaluating the stability and efficiency of various network paths through automated batch testing. It utilizes a concurrent worker system that executes isolated subprocesses to ensure individual node tests do not interfere wit