Awesome GitHub RepositoriesDistributed Task Orchestrators

Ray is a distributed computing framework designed to scale Python and Java applications across clusters by abstracting task scheduling and resource management. It functions as a resource-aware execution engine that manages task dependencies, placement, and fault tolerance across networked compute nodes. At its core, the system provides a stateful actor model, allowing developers to define classes that run in dedicated processes to maintain and mutate internal state across remote method calls. The framework distinguishes itself through a robust cross-language interoperability layer, enabling f

Conductor is a durable workflow engine designed to orchestrate complex, long-running business processes and autonomous agent loops. It functions as a stateful execution platform that persists the entire history of a process, ensuring that workflows remain reliable and recoverable across infrastructure failures, system restarts, and transient network errors. By managing task lifecycles, worker polling, and state transitions, it provides a centralized coordination layer for distributed systems. The platform distinguishes itself through its specialized support for AI agent orchestration, allowin

xxl-job is a distributed task scheduling platform and job orchestrator designed to manage and trigger timed jobs across a cluster of remote executor nodes. It provides a centralized system for scheduling tasks, linking dependent jobs, and managing complex execution lifecycles through a relational database that persists configurations and logs. The platform distinguishes itself through a web-based interface for cron job management, allowing users to create and update scheduled tasks without modifying source code. It supports cross-language task execution by triggering logic on third-party exec

This project is a build orchestration engine and development toolkit designed for managing large-scale monorepos. It provides a unified workspace environment that maps project relationships and dependencies, enabling the system to perform intelligent impact analysis and execute only the tasks affected by specific code changes. The system distinguishes itself through a persistent daemon that monitors file changes for near-instant feedback and a content-addressable caching mechanism that stores task outputs to prevent redundant computation across local and remote environments. It further suppor

Prefect is a workflow orchestration platform designed to define, schedule, and monitor complex data pipelines as Python code. It functions as a container-native engine that wraps individual tasks in isolated environments, ensuring consistent dependencies and resource allocation across diverse infrastructure. By utilizing a state-machine-based orchestration model, the system tracks execution progress through discrete transitions and persistent event logs to maintain reliable and observable task processing. The platform distinguishes itself through a decoupled worker-API architecture, which sep

SWE-agent is an autonomous software engineering platform designed to automate repository maintenance and issue resolution. By orchestrating language models to navigate codebases, diagnose software bugs, and apply fixes, the framework functions as an autonomous agent capable of executing shell commands, editing source code, and managing pull requests within isolated, containerized environments. The platform distinguishes itself through its focus on end-to-end task autonomy and observability. It features a robust trajectory logging system that records every thought, action, and environment obse

Chaos Monkey is a chaos engineering tool designed to verify the resilience of distributed systems by intentionally terminating production instances. It functions as a fault injection service that identifies weaknesses in cloud-based architectures by simulating real-world hardware and software outages. The platform operates through a centralized orchestration engine that executes periodic disruption cycles based on predefined configuration rules. It employs a rule-based selection process that evaluates instance metadata against safety constraints to ensure that only eligible targets are disrup

Argo Workflows is a container-native workflow engine that functions as a Kubernetes custom resource controller. It orchestrates complex sequences of containerized tasks by executing them as directed acyclic graphs, allowing for dependency management and parallel processing within a cluster. The system extends the native Kubernetes control plane to manage the full lifecycle of automated processes, from initial triggering to final resource cleanup. The platform distinguishes itself through its controller-pattern reconciliation, which continuously monitors workflow states to align them with desi

Fabric is a command-line interface and framework designed to integrate artificial intelligence reasoning into shell-based workflows. It functions as an orchestration tool that connects local data pipelines to remote artificial intelligence services, allowing users to automate content analysis and complex reasoning tasks directly from the terminal. The project distinguishes itself through a modular architecture that treats prompt patterns as version-controlled, reusable logic stored on the local filesystem. By utilizing standard input and output streams, it enables users to chain these analyti

Dask هو إطار عمل للحوسبة المتوازية وجدول مهام موزع مصمم لتوسيع نطاق سير عمل علوم البيانات في Python من أجهزة فردية إلى مجموعات (clusters) كبيرة. يعمل كمدير موارد للمجموعة يقوم بتنسيق المنطق الحسابي من خلال تمثيل المهام وتبعياتها كرسوم بيانية موجهة غير دورية. تسمح هذه البنية للنظام بأتمتة توزيع أعباء العمل عبر الأجهزة المتاحة مع إدارة متطلبات التنفيذ المعقدة. يتميز المشروع بمحرك تقييم كسول يؤجل عمليات البيانات حتى يتم طلبها صراحة، مما يتيح تحسين الرسم البياني العالمي وتخصيص الموارد بكفاءة. يتضمن خاصية تسريب البيانات الواعية بالذاكرة لمنع تعطل النظام عند معالجة مجموعات البيانات التي تتجاوز الذاكرة المتاحة، ويستخدم دمج الرسم البياني للمهام لدمج تسلسلات العمليات في خطوات تنفيذ واحدة، مما يقلل من عبء الجدولة والاتصال بين العقد. توفر المنصة سطح قدرات شاملاً لتحليلات البيانات واسعة النطاق، بما في ذلك دعم التعلم الآلي الموزع، وتكامل الحوسبة عالية الأداء، ومعالجة البيانات المتوازية. توفر أدوات واسعة النطاق لإدارة دورة حياة المجموعة، وتوصيف الأداء، والمراقبة في الوقت الفعلي لتنفيذ المهام. يمكن للمستخدمين نشر هذه البيئات عبر بنية تحتية متنوعة، بما في ذلك الأجهزة المحلية، ومزودي السحابة، والأنظمة الحاوية، ومجموعات الحوسبة عالية الأداء.

The AWS Cloud Development Kit is an infrastructure-as-code framework that enables developers to define and provision cloud resources using familiar programming languages. By utilizing construct-based synthesis, it translates high-level, object-oriented code into declarative templates, allowing for the automated management of complex cloud environments through a centralized, code-driven control plane. The framework distinguishes itself through its ability to model infrastructure as a dependency-aware resource graph, ensuring that components are provisioned and updated in the correct order. It

This repository serves as a library of reference patterns and scripts for infrastructure automation and configuration management. It provides a collection of standardized examples designed to demonstrate how to define and maintain server environments as code, ensuring consistency across development, testing, and production stages. The project focuses on implementing infrastructure as code best practices by showcasing how to structure automation logic for complex deployments. These examples illustrate the use of declarative modeling to define desired system states, alongside modular task abstr

This project is an infrastructure as code framework and library of reusable playbooks designed for server configuration and DevOps workflow automation. It provides a Linux server configuration suite and specialized tools for provisioning multi-node Kubernetes clusters to support containerized applications. The library enables the automation of infrastructure tasks and the orchestration of multi-server workflows. It includes specific logic for deploying containerized workloads and managing application environments across different hosting platforms. The codebase covers broad capability areas

Otter is a distributed database synchronization system and change data capture tool designed to replicate data between databases across multiple geographic regions. It functions as a synchronization orchestrator and ETL data pipeline that mirrors records and associated files in real time. The system employs incremental log parsing to capture database changes and utilizes a consistency-based convergence algorithm and loop-avoidance logic to manage bi-directional replication. It processes data through a pipeline of selection, extraction, transformation, and loading to handle joins and format co

Inngest is a durable execution framework and event-driven automation engine designed to orchestrate background workflows. It enables developers to build resilient, stateful processes by memoizing function steps, ensuring that long-running tasks can automatically resume from the last successful operation after failures, timeouts, or infrastructure restarts. The platform distinguishes itself through its event-driven architecture, which uses a schema-validated bus to trigger functions and coordinate complex, multi-step logic. It employs an onion-model middleware approach for cross-cutting concer

SparkInternals is a technical reference and architecture guide detailing the internal design and implementation of the Apache Spark distributed computing engine. It serves as a study of big data engine analysis, focusing on how the system manages cluster execution and the interaction between driver nodes, executors, and workers. The project provides a detailed breakdown of how logical plans are converted into physical execution stages. It specifically analyzes the mechanics of data shuffle operations, memory management, and the coordination of distributed job scheduling. The documentation co

Apfell هو إطار عمل للفريق الأحمر (red teaming) وخادم للقيادة والتحكم مصمم لمحاكاة الخصم التعاونية. يوفر بنية تحتية مركزية لإدارة الوكلاء البعيدين وتوزيع المهام عبر أنظمة تشغيل متعددة باستخدام وسيط رسائل للمزامنة في الوقت الفعلي. يعمل النظام كمنسق وكيل موزع، مما يسمح للفرق بتنسيق سلاسل هجوم معقدة ومزامنة بيانات الحاويات. يتميز بمدير حمولات متعدد المنصات يتيح تنزيل ودمج الوكلاء المخصصين وملفات تعريف الأوامر من مستودعات بعيدة. تغطي المنصة إدارة محاكاة الخصم، والتحكم الموزع في الأوامر، واستخدام ملفات تعريف الأوامر المعيارية للحفاظ على سلوكيات تنفيذ متسقة عبر بيئات مستهدفة مختلفة.

Chronos is a distributed, fault-tolerant job scheduler designed for managing containerized workloads within a cluster. It functions as a task orchestrator that automates the execution of recurring background jobs and complex, multi-step workflows across distributed computing resources. The system distinguishes itself through its ability to manage directed acyclic graph dependencies, ensuring that tasks are triggered only upon the successful completion of prerequisite jobs. It utilizes a leader-follower consensus architecture to maintain high availability and state persistence, while relying o

WebODM هو مجموعة برمجيات للمساحة التصويرية (photogrammetry) ومنصة لمعالجة صور الطائرات بدون طيار تُستخدم لتحويل الصور الجوية الخام إلى بيانات مكانية. تعمل كأداة خرائط جغرافية مكانية لتوليد خرائط ثنائية الأبعاد مرجعية جغرافياً، وسحب نقطية ثلاثية الأبعاد، ونماذج ارتفاع رقمية. يعمل النظام كمولد شبكات ثلاثية الأبعاد (3D mesh) يحول الصور الجوية المتداخلة إلى نماذج ذات ملمس. يوفر سير عمل للخرائط الجوية لإنتاج مخرجات دقيقة للتحليل الجغرافي والمسح من خلال محركات مساحة تصويرية متنوعة. تتضمن المنصة قدرات لتصور البيانات الجغرافية المكانية ومعالجة الصور الجوية. وتستخدم خط أنابيب معالجة لتوحيد سير العمل عبر خلفيات مختلفة وتتعامل مع تحويل الصور إلى تنسيقات مرجعية جغرافياً موحدة لتطبيقات الخرائط ونظم المعلومات الجغرافية (GIS).

Seneca هو إطار عمل بنية قائمة على الرسائل ومجموعة أدوات للخدمات المصغرة لـ Node.js. يعمل كمنسق مهام موزع وموجه رسائل قائم على الأنماط، مما يسمح للمطورين ببناء أنظمة من الخدمات المنفصلة التي تتواصل عبر ناقل رسائل. يتميز إطار العمل بنظام إضافات معياري ينظم منطق العمل في وحدات قابلة لإعادة الاستخدام وقابلة للتكوين. ويدعم امتدادات الإجراءات الديناميكية، مما يتيح للمعالجات الجديدة تغليف أو تجاوز أنماط الإجراءات الحالية لحقن منطق مخصص دون تعديل الكود الأصلي. يغطي النظام مجموعة واسعة من القدرات بما في ذلك اتصالات الخدمة الموزعة، وتنسيق الإجراءات غير المتزامن، وإدارة بيانات الكيانات. كما يوفر أدوات مراقبة لتتبع أنماط الإجراءات، وتصور السجلات المستند إلى الويب، وأدوات ضمان الجودة لمحاكاة تبعيات الخدمة.