Awesome GitHub RepositoriesDistributed Task Orchestrators

Ray is a distributed computing framework designed to scale Python and Java applications across clusters by abstracting task scheduling and resource management. It functions as a resource-aware execution engine that manages task dependencies, placement, and fault tolerance across networked compute nodes. At its core, the system provides a stateful actor model, allowing developers to define classes that run in dedicated processes to maintain and mutate internal state across remote method calls. The framework distinguishes itself through a robust cross-language interoperability layer, enabling f

Conductor is a durable workflow engine designed to orchestrate complex, long-running business processes and autonomous agent loops. It functions as a stateful execution platform that persists the entire history of a process, ensuring that workflows remain reliable and recoverable across infrastructure failures, system restarts, and transient network errors. By managing task lifecycles, worker polling, and state transitions, it provides a centralized coordination layer for distributed systems. The platform distinguishes itself through its specialized support for AI agent orchestration, allowin

xxl-job is a distributed task scheduling platform and job orchestrator designed to manage and trigger timed jobs across a cluster of remote executor nodes. It provides a centralized system for scheduling tasks, linking dependent jobs, and managing complex execution lifecycles through a relational database that persists configurations and logs. The platform distinguishes itself through a web-based interface for cron job management, allowing users to create and update scheduled tasks without modifying source code. It supports cross-language task execution by triggering logic on third-party exec

This project is a build orchestration engine and development toolkit designed for managing large-scale monorepos. It provides a unified workspace environment that maps project relationships and dependencies, enabling the system to perform intelligent impact analysis and execute only the tasks affected by specific code changes. The system distinguishes itself through a persistent daemon that monitors file changes for near-instant feedback and a content-addressable caching mechanism that stores task outputs to prevent redundant computation across local and remote environments. It further suppor

Prefect is a workflow orchestration platform designed to define, schedule, and monitor complex data pipelines as Python code. It functions as a container-native engine that wraps individual tasks in isolated environments, ensuring consistent dependencies and resource allocation across diverse infrastructure. By utilizing a state-machine-based orchestration model, the system tracks execution progress through discrete transitions and persistent event logs to maintain reliable and observable task processing. The platform distinguishes itself through a decoupled worker-API architecture, which sep

SWE-agent is an autonomous software engineering platform designed to automate repository maintenance and issue resolution. By orchestrating language models to navigate codebases, diagnose software bugs, and apply fixes, the framework functions as an autonomous agent capable of executing shell commands, editing source code, and managing pull requests within isolated, containerized environments. The platform distinguishes itself through its focus on end-to-end task autonomy and observability. It features a robust trajectory logging system that records every thought, action, and environment obse

Chaos Monkey is a chaos engineering tool designed to verify the resilience of distributed systems by intentionally terminating production instances. It functions as a fault injection service that identifies weaknesses in cloud-based architectures by simulating real-world hardware and software outages. The platform operates through a centralized orchestration engine that executes periodic disruption cycles based on predefined configuration rules. It employs a rule-based selection process that evaluates instance metadata against safety constraints to ensure that only eligible targets are disrup

Argo Workflows is a container-native workflow engine that functions as a Kubernetes custom resource controller. It orchestrates complex sequences of containerized tasks by executing them as directed acyclic graphs, allowing for dependency management and parallel processing within a cluster. The system extends the native Kubernetes control plane to manage the full lifecycle of automated processes, from initial triggering to final resource cleanup. The platform distinguishes itself through its controller-pattern reconciliation, which continuously monitors workflow states to align them with desi

Fabric is a command-line interface and framework designed to integrate artificial intelligence reasoning into shell-based workflows. It functions as an orchestration tool that connects local data pipelines to remote artificial intelligence services, allowing users to automate content analysis and complex reasoning tasks directly from the terminal. The project distinguishes itself through a modular architecture that treats prompt patterns as version-controlled, reusable logic stored on the local filesystem. By utilizing standard input and output streams, it enables users to chain these analyti

Dask este un framework de calcul paralel și un scheduler de sarcini distribuit conceput pentru a scala fluxurile de lucru de știința datelor în Python de la mașini individuale la clustere mari. Acesta funcționează ca un manager de resurse de cluster care orchestrează logica computațională prin reprezentarea sarcinilor și a dependențelor acestora sub formă de grafuri aciclice direcționate. Această arhitectură permite sistemului să automatizeze distribuția sarcinilor de lucru pe hardware-ul disponibil, gestionând în același timp cerințe complexe de execuție. Proiectul se distinge printr-un motor de evaluare leneșă (lazy) care amână operațiunile pe date până când sunt solicitate explicit, permițând optimizarea globală a grafului și alocarea eficientă a resurselor. Acesta încorporează „spilling” de date conștient de memorie pentru a preveni blocarea sistemului la procesarea seturilor de date care depășesc memoria disponibilă și utilizează fuziunea grafului de sarcini pentru a combina secvențe de operațiuni în pași de execuție unici, minimizând overhead-ul de programare și comunicarea între noduri. Platforma oferă o suprafață cuprinzătoare de capabilități pentru analiza datelor la scară largă, inclusiv suport pentru învățare automată distribuită, integrare cu calcul de înaltă performanță și procesare paralelă a datelor. Oferă instrumente extinse pentru gestionarea ciclului de viață al clusterului, profilarea performanței și monitorizarea în timp real a execuției sarcinilor. Utilizatorii pot implementa aceste medii pe diverse infrastructuri, inclusiv hardware local, furnizori de cloud, sisteme containerizate și clustere de calcul de înaltă performanță.

The AWS Cloud Development Kit is an infrastructure-as-code framework that enables developers to define and provision cloud resources using familiar programming languages. By utilizing construct-based synthesis, it translates high-level, object-oriented code into declarative templates, allowing for the automated management of complex cloud environments through a centralized, code-driven control plane. The framework distinguishes itself through its ability to model infrastructure as a dependency-aware resource graph, ensuring that components are provisioned and updated in the correct order. It

This repository serves as a library of reference patterns and scripts for infrastructure automation and configuration management. It provides a collection of standardized examples designed to demonstrate how to define and maintain server environments as code, ensuring consistency across development, testing, and production stages. The project focuses on implementing infrastructure as code best practices by showcasing how to structure automation logic for complex deployments. These examples illustrate the use of declarative modeling to define desired system states, alongside modular task abstr

This project is an infrastructure as code framework and library of reusable playbooks designed for server configuration and DevOps workflow automation. It provides a Linux server configuration suite and specialized tools for provisioning multi-node Kubernetes clusters to support containerized applications. The library enables the automation of infrastructure tasks and the orchestration of multi-server workflows. It includes specific logic for deploying containerized workloads and managing application environments across different hosting platforms. The codebase covers broad capability areas

Otter is a distributed database synchronization system and change data capture tool designed to replicate data between databases across multiple geographic regions. It functions as a synchronization orchestrator and ETL data pipeline that mirrors records and associated files in real time. The system employs incremental log parsing to capture database changes and utilizes a consistency-based convergence algorithm and loop-avoidance logic to manage bi-directional replication. It processes data through a pipeline of selection, extraction, transformation, and loading to handle joins and format co

Inngest is a durable execution framework and event-driven automation engine designed to orchestrate background workflows. It enables developers to build resilient, stateful processes by memoizing function steps, ensuring that long-running tasks can automatically resume from the last successful operation after failures, timeouts, or infrastructure restarts. The platform distinguishes itself through its event-driven architecture, which uses a schema-validated bus to trigger functions and coordinate complex, multi-step logic. It employs an onion-model middleware approach for cross-cutting concer

SparkInternals este un ghid tehnic de referință și arhitectură care detaliază designul intern și implementarea motorului de calcul distribuit Apache Spark. Acesta servește drept studiu de analiză a motoarelor de big data, concentrându-se pe modul în care sistemul gestionează execuția în cluster și interacțiunea dintre nodurile driver, executori și workeri. Proiectul oferă o detaliere a modului în care planurile logice sunt convertite în etape de execuție fizică. Analizează în mod specific mecanica operațiunilor de shuffle a datelor, gestionarea memoriei și coordonarea programării joburilor distribuite. Documentația acoperă o gamă largă de capabilități de calcul distribuit, inclusiv planificarea execuției interogărilor, gestionarea dependențelor de date și strategii de caching în memorie. De asemenea, examinează distribuția sarcinilor, execuția paralelă și procesele utilizate pentru recuperarea în caz de eroare și persistența datelor.

Apfell este un framework de red teaming și un server de comandă și control (C2) conceput pentru simularea colaborativă a adversarilor. Oferă o infrastructură centralizată pentru a gestiona agenții la distanță și a distribui sarcinile pe mai multe sisteme de operare, folosind un message broker pentru sincronizare în timp real. Sistemul funcționează ca un orchestrator distribuit de agenți, permițând echipelor să coordoneze lanțuri complexe de atac și să sincronizeze datele containerelor. Dispune de un manager de payload-uri multi-platformă care permite descărcarea și integrarea agenților personalizați și a profilurilor de comenzi din repository-uri la distanță. Platforma acoperă gestionarea simulărilor de adversari, controlul distribuit al comenzilor și utilizarea profilării modulare a comenzilor pentru a menține comportamente de execuție consistente pe diferite medii țintă.

Chronos is a distributed, fault-tolerant job scheduler designed for managing containerized workloads within a cluster. It functions as a task orchestrator that automates the execution of recurring background jobs and complex, multi-step workflows across distributed computing resources. The system distinguishes itself through its ability to manage directed acyclic graph dependencies, ensuring that tasks are triggered only upon the successful completion of prerequisite jobs. It utilizes a leader-follower consensus architecture to maintain high availability and state persistence, while relying o

WebODM is a photogrammetry software suite and drone image processing platform used to transform raw aerial photographs into spatial data. It functions as a geospatial mapping tool for generating georeferenced 2D maps, 3D point clouds, and digital elevation models. The system operates as a 3D mesh generator that converts overlapping aerial photos into textured models. It provides an aerial mapping workflow to produce precise outputs for geographic analysis and surveying through various photogrammetry engines. The platform includes capabilities for geospatial data visualization and the process

Seneca is a message-driven architecture framework and microservices toolkit for Node.js. It functions as a distributed task orchestrator and pattern-based message router, allowing developers to build systems of decoupled services that communicate via a message bus. The framework distinguishes itself through a modular plugin system that organizes business logic into reusable, configurable modules. It supports dynamic action extensions, enabling new handlers to wrap or override existing action patterns to inject custom logic without modifying original code. The system covers a broad range of c