Awesome GitHub RepositoriesDistributed Job Executors

xxl-job is a distributed task scheduling platform and job orchestrator designed to manage and trigger timed jobs across a cluster of remote executor nodes. It provides a centralized system for scheduling tasks, linking dependent jobs, and managing complex execution lifecycles through a relational database that persists configurations and logs. The platform distinguishes itself through a web-based interface for cron job management, allowing users to create and update scheduled tasks without modifying source code. It supports cross-language task execution by triggering logic on third-party exec

DataX Web is a web-based management platform for scheduling, building, executing, and monitoring distributed data synchronization jobs powered by DataX. It provides a visual console for creating and managing DataX tasks without manual JSON configuration, with a distributed executor cluster that auto-registers worker nodes and supports configurable routing and blocking strategies for task distribution. The platform offers cron-based task scheduling with dynamic start, stop, and immediate status changes, along with incremental sync capabilities that pass dynamic parameters to extract only new o

Azkaban هو مدير سير عمل موزع ومنسق وظائف قائم على DAG مصمم كمعالج دفعات للمؤسسات. يعمل كمحرك سير عمل قائم على Java يقوم بجدولة وتنفيذ تسلسلات وظائف معقدة عبر مجموعة من خوادم التنفيذ، مع وظائف محددة لإدارة أحمال عمل البيانات الضخمة على عناقيد Hadoop. يتميز النظام بنموذج تنفيذ موزع ينسق الحالة عبر قاعدة بيانات مشتركة لضمان التوفر العالي. يستخدم بنية قائمة على الإضافات تسمح بأنواع وظائف مخصصة وامتدادات وظائف النظام، بما في ذلك القدرة على إعادة تحميل الإضافات دون إعادة تشغيل خوادم التنفيذ. تغطي المنصة مجموعة واسعة من القدرات بما في ذلك تنسيق خط أنابيب البيانات مع المنطق الشرطي، والجدولة الدورية والقائمة على الأحداث، ومراقبة المؤسسة مع تتبع SLA. يوفر تحكماً دقيقاً في الوصول وانتحال هوية المستخدم للتنفيذ الآمن، إلى جانب أدوات إدارة حركة المرور لموازنة تحميل التنفيذ وحصص الموارد. يمكن للمستخدمين إدارة سير العمل من خلال واجهة قائمة على الويب أو برمجياً عبر API تنفيذ سير العمل.

Slurm is a cluster workload manager and job scheduler designed for high-performance computing environments. It functions as a distributed compute orchestrator that queues and executes large-scale computational tasks across multiple compute nodes in a cluster. The system acts as a resource arbitrator, distributing hardware nodes and processors among concurrent users to prevent resource conflicts and maximize efficiency. It coordinates the simultaneous launch of multiple processes across different physical servers to execute parallel jobs and scientific workloads. The platform covers broad cap

This project provides a containerized environment for deploying Apache Airflow, enabling the orchestration of complex data pipelines and automated task scheduling. By packaging the orchestration platform into portable images, it ensures consistent execution across diverse infrastructure setups and simplifies the management of runtime dependencies. The platform facilitates distributed task execution by decoupling the scheduler from the execution layer, allowing for horizontal scaling of processing power across multiple worker nodes. It supports dynamic configuration through environment variabl

proxy.py is a lightweight, dependency-free HTTP and reverse proxy framework built in Python that handles network traffic through asynchronous event loops and multi-core connection distribution. It acts as a customizable proxy server and embedded web server, allowing developers to inspect, route, and modify web traffic programmatically or via command-line execution wrappers. The platform provides in-flight TLS interception and traffic decryption using generated root certificates, alongside modular extension hooks that support custom dashboard interfaces, request handling plugins, and REST API

Gravitino is a federated metadata lake and unified data catalog designed to manage tables, files, and AI models across diverse data sources and cloud storage. It serves as a centralized interface for governing schemas, access controls, and tagging across relational databases, messaging queues, and object stores. The project distinguishes itself by unifying the management of AI assets, such as machine learning models and their version lineages, alongside traditional tabular data. It also implements the Iceberg REST specification to provide a standardized metadata server and proxy for lakehouse