Awesome GitHub RepositoriesDistributed Task Orchestrators

Ray is a distributed computing framework designed to scale Python and Java applications across clusters by abstracting task scheduling and resource management. It functions as a resource-aware execution engine that manages task dependencies, placement, and fault tolerance across networked compute nodes. At its core, the system provides a stateful actor model, allowing developers to define classes that run in dedicated processes to maintain and mutate internal state across remote method calls. The framework distinguishes itself through a robust cross-language interoperability layer, enabling f

Conductor is a durable workflow engine designed to orchestrate complex, long-running business processes and autonomous agent loops. It functions as a stateful execution platform that persists the entire history of a process, ensuring that workflows remain reliable and recoverable across infrastructure failures, system restarts, and transient network errors. By managing task lifecycles, worker polling, and state transitions, it provides a centralized coordination layer for distributed systems. The platform distinguishes itself through its specialized support for AI agent orchestration, allowin

xxl-job is a distributed task scheduling platform and job orchestrator designed to manage and trigger timed jobs across a cluster of remote executor nodes. It provides a centralized system for scheduling tasks, linking dependent jobs, and managing complex execution lifecycles through a relational database that persists configurations and logs. The platform distinguishes itself through a web-based interface for cron job management, allowing users to create and update scheduled tasks without modifying source code. It supports cross-language task execution by triggering logic on third-party exec

This project is a build orchestration engine and development toolkit designed for managing large-scale monorepos. It provides a unified workspace environment that maps project relationships and dependencies, enabling the system to perform intelligent impact analysis and execute only the tasks affected by specific code changes. The system distinguishes itself through a persistent daemon that monitors file changes for near-instant feedback and a content-addressable caching mechanism that stores task outputs to prevent redundant computation across local and remote environments. It further suppor

Prefect is a workflow orchestration platform designed to define, schedule, and monitor complex data pipelines as Python code. It functions as a container-native engine that wraps individual tasks in isolated environments, ensuring consistent dependencies and resource allocation across diverse infrastructure. By utilizing a state-machine-based orchestration model, the system tracks execution progress through discrete transitions and persistent event logs to maintain reliable and observable task processing. The platform distinguishes itself through a decoupled worker-API architecture, which sep

SWE-agent is an autonomous software engineering platform designed to automate repository maintenance and issue resolution. By orchestrating language models to navigate codebases, diagnose software bugs, and apply fixes, the framework functions as an autonomous agent capable of executing shell commands, editing source code, and managing pull requests within isolated, containerized environments. The platform distinguishes itself through its focus on end-to-end task autonomy and observability. It features a robust trajectory logging system that records every thought, action, and environment obse

Chaos Monkey is a chaos engineering tool designed to verify the resilience of distributed systems by intentionally terminating production instances. It functions as a fault injection service that identifies weaknesses in cloud-based architectures by simulating real-world hardware and software outages. The platform operates through a centralized orchestration engine that executes periodic disruption cycles based on predefined configuration rules. It employs a rule-based selection process that evaluates instance metadata against safety constraints to ensure that only eligible targets are disrup

Argo Workflows is a container-native workflow engine that functions as a Kubernetes custom resource controller. It orchestrates complex sequences of containerized tasks by executing them as directed acyclic graphs, allowing for dependency management and parallel processing within a cluster. The system extends the native Kubernetes control plane to manage the full lifecycle of automated processes, from initial triggering to final resource cleanup. The platform distinguishes itself through its controller-pattern reconciliation, which continuously monitors workflow states to align them with desi

Fabric is a command-line interface and framework designed to integrate artificial intelligence reasoning into shell-based workflows. It functions as an orchestration tool that connects local data pipelines to remote artificial intelligence services, allowing users to automate content analysis and complex reasoning tasks directly from the terminal. The project distinguishes itself through a modular architecture that treats prompt patterns as version-controlled, reusable logic stored on the local filesystem. By utilizing standard input and output streams, it enables users to chain these analyti

Dask एक पैरेलल कंप्यूटिंग फ्रेमवर्क और डिस्ट्रीब्यूटेड टास्क शेड्यूलर है जिसे Python डेटा साइंस वर्कफ़्लो को सिंगल मशीनों से बड़े क्लस्टर्स तक स्केल करने के लिए डिज़ाइन किया गया है। यह एक क्लस्टर रिसोर्स मैनेजर के रूप में कार्य करता है जो कार्यों और उनकी डिपेंडेंसी को डायरेक्टेड एसाइक्लिक ग्राफ (DAGs) के रूप में प्रस्तुत करके कम्प्यूटेशनल लॉजिक को व्यवस्थित करता है। यह आर्किटेक्चर सिस्टम को जटिल निष्पादन आवश्यकताओं का प्रबंधन करते हुए उपलब्ध हार्डवेयर पर वर्कलोड के वितरण को स्वचालित करने की अनुमति देता है। यह प्रोजेक्ट एक लेज़ी इवैल्यूएशन इंजन के माध्यम से खुद को अलग करता है जो डेटा ऑपरेशन्स को तब तक स्थगित कर देता है जब तक कि उन्हें स्पष्ट रूप से अनुरोध न किया जाए, जिससे ग्लोबल ग्राफ ऑप्टिमाइज़ेशन और कुशल संसाधन आवंटन सक्षम होता है। इसमें उपलब्ध मेमोरी से अधिक डेटासेट को प्रोसेस करते समय सिस्टम क्रैश को रोकने के लिए मेमोरी-अवेयर डेटा स्पिलिंग शामिल है, और यह टास्क ग्राफ फ्यूजन का उपयोग ऑपरेशन्स के अनुक्रमों को एकल निष्पादन चरणों में संयोजित करने के लिए करता है, जिससे शेड्यूलिंग ओवरहेड और इंटर-नोड संचार कम हो जाता है। यह प्लेटफॉर्म बड़े पैमाने पर डेटा एनालिटिक्स के लिए एक व्यापक क्षमता सतह प्रदान करता है, जिसमें डिस्ट्रीब्यूटेड मशीन लर्निंग, उच्च-प्रदर्शन कंप्यूटिंग एकीकरण, और पैरेलल डेटा प्रोसेसिंग के लिए समर्थन शामिल है। यह क्लस्टर लाइफसाइकिल मैनेजमेंट, परफॉरमेंस प्रोफाइलिंग, और टास्क निष्पादन की रीयल-टाइम मॉनिटरिंग के लिए व्यापक उपकरण प्रदान करता है। उपयोगकर्ता इन वातावरणों को स्थानीय हार्डवेयर, क्लाउड प्रदाताओं, कंटेनरीकृत सिस्टम, और उच्च-प्रदर्शन कंप्यूटिंग क्लस्टर्स सहित विविध बुनियादी ढांचे पर तैनात कर सकते हैं।

The AWS Cloud Development Kit is an infrastructure-as-code framework that enables developers to define and provision cloud resources using familiar programming languages. By utilizing construct-based synthesis, it translates high-level, object-oriented code into declarative templates, allowing for the automated management of complex cloud environments through a centralized, code-driven control plane. The framework distinguishes itself through its ability to model infrastructure as a dependency-aware resource graph, ensuring that components are provisioned and updated in the correct order. It

This repository serves as a library of reference patterns and scripts for infrastructure automation and configuration management. It provides a collection of standardized examples designed to demonstrate how to define and maintain server environments as code, ensuring consistency across development, testing, and production stages. The project focuses on implementing infrastructure as code best practices by showcasing how to structure automation logic for complex deployments. These examples illustrate the use of declarative modeling to define desired system states, alongside modular task abstr

This project is an infrastructure as code framework and library of reusable playbooks designed for server configuration and DevOps workflow automation. It provides a Linux server configuration suite and specialized tools for provisioning multi-node Kubernetes clusters to support containerized applications. The library enables the automation of infrastructure tasks and the orchestration of multi-server workflows. It includes specific logic for deploying containerized workloads and managing application environments across different hosting platforms. The codebase covers broad capability areas

Otter is a distributed database synchronization system and change data capture tool designed to replicate data between databases across multiple geographic regions. It functions as a synchronization orchestrator and ETL data pipeline that mirrors records and associated files in real time. The system employs incremental log parsing to capture database changes and utilizes a consistency-based convergence algorithm and loop-avoidance logic to manage bi-directional replication. It processes data through a pipeline of selection, extraction, transformation, and loading to handle joins and format co

Inngest is a durable execution framework and event-driven automation engine designed to orchestrate background workflows. It enables developers to build resilient, stateful processes by memoizing function steps, ensuring that long-running tasks can automatically resume from the last successful operation after failures, timeouts, or infrastructure restarts. The platform distinguishes itself through its event-driven architecture, which uses a schema-validated bus to trigger functions and coordinate complex, multi-step logic. It employs an onion-model middleware approach for cross-cutting concer

SparkInternals is a technical reference and architecture guide detailing the internal design and implementation of the Apache Spark distributed computing engine. It serves as a study of big data engine analysis, focusing on how the system manages cluster execution and the interaction between driver nodes, executors, and workers. The project provides a detailed breakdown of how logical plans are converted into physical execution stages. It specifically analyzes the mechanics of data shuffle operations, memory management, and the coordination of distributed job scheduling. The documentation co

Apfell is a red teaming framework and command and control server designed for collaborative adversary simulation. It provides a centralized infrastructure to manage remote agents and distribute tasking across multiple operating systems using a message broker for real-time synchronization. The system functions as a distributed agent orchestrator, allowing teams to coordinate complex attack chains and synchronize container data. It features a multi-platform payload manager that enables the downloading and integration of custom agents and command profiles from remote repositories. The platform

Chronos is a distributed, fault-tolerant job scheduler designed for managing containerized workloads within a cluster. It functions as a task orchestrator that automates the execution of recurring background jobs and complex, multi-step workflows across distributed computing resources. The system distinguishes itself through its ability to manage directed acyclic graph dependencies, ensuring that tasks are triggered only upon the successful completion of prerequisite jobs. It utilizes a leader-follower consensus architecture to maintain high availability and state persistence, while relying o

WebODM एक फ़ोटोग्रामेट्री सॉफ़्टवेयर सुइट और ड्रोन इमेज प्रोसेसिंग प्लेटफ़ॉर्म है जिसका उपयोग कच्ची हवाई तस्वीरों को स्थानिक डेटा में बदलने के लिए किया जाता है। यह जियोरेफरेंस्ड 2D मैप्स, 3D पॉइंट क्लाउड और डिजिटल एलिवेशन मॉडल उत्पन्न करने के लिए एक भू-स्थानिक मैपिंग टूल के रूप में कार्य करता है। यह सिस्टम एक 3D मेश जनरेटर के रूप में संचालित होता है जो ओवरलैपिंग हवाई तस्वीरों को टेक्सचर्ड मॉडल में बदल देता है। यह विभिन्न फ़ोटोग्रामेट्री इंजनों के माध्यम से भौगोलिक विश्लेषण और सर्वेक्षण के लिए सटीक आउटपुट तैयार करने के लिए एक हवाई मैपिंग वर्कफ़्लो प्रदान करता है। इस प्लेटफ़ॉर्म में भू-स्थानिक डेटा विज़ुअलाइज़ेशन और हवाई इमेजरी के प्रसंस्करण के लिए क्षमताएं शामिल हैं। यह विभिन्न बैकएंड्स में वर्कफ़्लो को मानकीकृत करने के लिए एक प्रोसेसिंग पाइपलाइन का उपयोग करता है और मैपिंग और GIS एप्लिकेशन के लिए इमेजरी को मानकीकृत जियोरेफरेंस्ड फॉर्मेट्स में बदलने का कार्य करता है।

Seneca is a message-driven architecture framework and microservices toolkit for Node.js. It functions as a distributed task orchestrator and pattern-based message router, allowing developers to build systems of decoupled services that communicate via a message bus. The framework distinguishes itself through a modular plugin system that organizes business logic into reusable, configurable modules. It supports dynamic action extensions, enabling new handlers to wrap or override existing action patterns to inject custom logic without modifying original code. The system covers a broad range of c