Awesome GitHub RepositoriesDistributed Task Schedulers

Airflow is a platform for programmatically authoring, scheduling, and monitoring complex data pipelines. It functions as a workflow automation engine that manages the lifecycle of recurring business processes by executing code-defined task dependencies. By representing workflows as directed acyclic graphs, the system ensures that task execution order and data flow are explicitly defined and reliably maintained across distributed computing environments. The platform distinguishes itself through a highly modular, provider-based architecture that decouples core orchestration logic from external

This project is a comprehensive reference collection of practical implementation examples and patterns for building applications with Spring Boot. It serves as a Java web application template and a showcase for developing functional web services featuring REST endpoints, template engines, and global exception handling. The repository distinguishes itself by providing detailed demonstrations of enterprise-grade features, including distributed locking, task scheduling, and asynchronous message exchange using brokers like RabbitMQ. It also includes reference implementations for automated API doc

Osquery is a unified endpoint monitoring framework that exposes operating system internals as relational tables. By representing hardware, network, and process activity as structured data, it allows users to retrieve system state and configuration information using standard SQL syntax. The system distinguishes itself through a cross-platform abstraction layer that normalizes disparate operating system interfaces into a consistent schema across Windows, macOS, and Linux. It supports both interactive local analysis via a command-line shell and distributed fleet orchestration, where recurring qu

Prefect is a workflow orchestration platform designed to define, schedule, and monitor complex data pipelines as Python code. It functions as a container-native engine that wraps individual tasks in isolated environments, ensuring consistent dependencies and resource allocation across diverse infrastructure. By utilizing a state-machine-based orchestration model, the system tracks execution progress through discrete transitions and persistent event logs to maintain reliable and observable task processing. The platform distinguishes itself through a decoupled worker-API architecture, which sep

Luigi is a Python framework designed for building and managing complex batch data pipelines. It functions as a workflow orchestration engine that organizes tasks into directed acyclic graphs, ensuring that jobs execute in the correct logical order based on their dependencies. By utilizing a centralized scheduler, the system coordinates task execution across distributed environments, tracks global workflow state, and prevents redundant processing by verifying the existence of output targets before triggering any work. The project distinguishes itself through a robust state-tracking mechanism t

This project is a collection of educational resources and reference implementations for the Apache Flink stream processing framework. It provides a learning resource focused on mastering distributed stream processing through implementation guides, performance tuning tutorials, and practical examples. The repository features detailed walkthroughs for building real-time data pipelines using the DataStream and Table APIs. It includes specific integration examples for connecting Apache Flink with Kafka brokers and Elasticsearch indices, as well as reference implementations for real-time deduplica

DolphinScheduler is a distributed workflow orchestrator designed to manage and automate complex data processing pipelines. It functions as a data pipeline scheduler that coordinates multi-step tasks across distributed environments, ensuring reliable execution through defined dependencies and sequences. The platform utilizes a directed acyclic graph model to represent workflows, allowing users to define task relationships via a visual interface. It employs a master-worker architecture supported by a pluggable task plugin system, which enables the dynamic extension of task types without requiri

Dask एक पैरेलल कंप्यूटिंग फ्रेमवर्क और डिस्ट्रीब्यूटेड टास्क शेड्यूलर है जिसे Python डेटा साइंस वर्कफ़्लो को सिंगल मशीनों से बड़े क्लस्टर्स तक स्केल करने के लिए डिज़ाइन किया गया है। यह एक क्लस्टर रिसोर्स मैनेजर के रूप में कार्य करता है जो कार्यों और उनकी डिपेंडेंसी को डायरेक्टेड एसाइक्लिक ग्राफ (DAGs) के रूप में प्रस्तुत करके कम्प्यूटेशनल लॉजिक को व्यवस्थित करता है। यह आर्किटेक्चर सिस्टम को जटिल निष्पादन आवश्यकताओं का प्रबंधन करते हुए उपलब्ध हार्डवेयर पर वर्कलोड के वितरण को स्वचालित करने की अनुमति देता है। यह प्रोजेक्ट एक लेज़ी इवैल्यूएशन इंजन के माध्यम से खुद को अलग करता है जो डेटा ऑपरेशन्स को तब तक स्थगित कर देता है जब तक कि उन्हें स्पष्ट रूप से अनुरोध न किया जाए, जिससे ग्लोबल ग्राफ ऑप्टिमाइज़ेशन और कुशल संसाधन आवंटन सक्षम होता है। इसमें उपलब्ध मेमोरी से अधिक डेटासेट को प्रोसेस करते समय सिस्टम क्रैश को रोकने के लिए मेमोरी-अवेयर डेटा स्पिलिंग शामिल है, और यह टास्क ग्राफ फ्यूजन का उपयोग ऑपरेशन्स के अनुक्रमों को एकल निष्पादन चरणों में संयोजित करने के लिए करता है, जिससे शेड्यूलिंग ओवरहेड और इंटर-नोड संचार कम हो जाता है। यह प्लेटफॉर्म बड़े पैमाने पर डेटा एनालिटिक्स के लिए एक व्यापक क्षमता सतह प्रदान करता है, जिसमें डिस्ट्रीब्यूटेड मशीन लर्निंग, उच्च-प्रदर्शन कंप्यूटिंग एकीकरण, और पैरेलल डेटा प्रोसेसिंग के लिए समर्थन शामिल है। यह क्लस्टर लाइफसाइकिल मैनेजमेंट, परफॉरमेंस प्रोफाइलिंग, और टास्क निष्पादन की रीयल-टाइम मॉनिटरिंग के लिए व्यापक उपकरण प्रदान करता है। उपयोगकर्ता इन वातावरणों को स्थानीय हार्डवेयर, क्लाउड प्रदाताओं, कंटेनरीकृत सिस्टम, और उच्च-प्रदर्शन कंप्यूटिंग क्लस्टर्स सहित विविध बुनियादी ढांचे पर तैनात कर सकते हैं।

Ydata-profiling is an automated exploratory data analysis framework designed to generate comprehensive statistical reports and visual summaries from dataframes. It functions as a diagnostic tool for assessing data quality, identifying missing values, duplicates, and outliers, while providing a scalable engine for profiling massive datasets across distributed enterprise environments. The project distinguishes itself through its ability to handle large-scale data through distributed task orchestration and lazy stream processing, which minimizes memory overhead during complex computations. It in

The AWS Cloud Development Kit is an infrastructure-as-code framework that enables developers to define and provision cloud resources using familiar programming languages. By utilizing construct-based synthesis, it translates high-level, object-oriented code into declarative templates, allowing for the automated management of complex cloud environments through a centralized, code-driven control plane. The framework distinguishes itself through its ability to model infrastructure as a dependency-aware resource graph, ensuring that components are provisioned and updated in the correct order. It

This project is a comprehensive learning resource and reference guide for software architecture and distributed systems design. It serves as a structured curriculum for engineers to study fundamental architectural patterns, scalability strategies, and distributed computing theory, specifically tailored to prepare for technical interviews and professional engineering roles. The repository distinguishes itself by providing a curated collection of industry-standard infrastructure tools and methodologies. It covers the selection and implementation of technologies for data storage, message brokeri

Boto3 is the AWS SDK for Python, providing a programmatic interface for managing and automating AWS cloud infrastructure and services. It serves as a cloud management API client and resource manager for provisioning, configuring, and scaling virtual servers, databases, and storage. The library enables the implementation of infrastructure-as-code through declarative templates and scripts, allowing for the deployment of identical resource stacks across multiple accounts and geographic regions. It also provides a framework for coordinating distributed workflows, serverless functions, and contain

ShardSphere-ElasticJob is a Java-based distributed scheduling framework designed to manage workloads across multiple nodes. It provides a system for splitting scheduled tasks into shards and distributing them across a cluster to achieve high-throughput execution. The framework includes a distributed task failover system that detects node failures and automatically reassigns missed job executions to healthy cluster instances. It also features a cluster resource manager to dynamically allocate execution resources based on system load and capacity. The system covers high-availability task execu

Enterprise job scheduling middleware with distributed computing ability.

Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

Cronicle is a distributed job scheduler that replaces traditional cron with a browser-based management interface. It runs scheduled tasks across a cluster of servers with automatic failover, using a custom cron parser that intersects day-of-month and day-of-week constraints when both are specified. The system executes jobs through a plugin framework that runs command-line scripts in any language, communicating via JSON over standard input and output. The scheduler provides a web-based real-time dashboard for monitoring running jobs with live logs, resource usage charts, and progress updates.

Scrapy-Redis is a library that transforms Scrapy into a distributed web crawling framework by replacing its in-memory scheduler with a Redis-backed component. This allows multiple Scrapy spider workers to coordinate through a shared request queue, enabling them to consume URLs concurrently while a Redis set tracks seen URLs across all workers to prevent duplicate crawls. The system persists crawl state—including pending requests and already-crawled URLs—in Redis, so a paused or crashed spider can resume from where it left off without losing progress. The library provides a Redis-based duplica

यह प्रोजेक्ट एक डिस्ट्रीब्यूटेड हेडलेस Chrome वेब क्रॉलर और डेटा एक्सट्रैक्शन फ्रेमवर्क है। यह एक JavaScript रेंडरिंग इंजन के रूप में कार्य करता है जो डायनामिक पेजों को संसाधित करने के लिए हेडलेस ब्राउज़र का उपयोग करता है, उन वेबसाइटों से संरचित डेटा निकालता है जिन्हें JavaScript निष्पादन की आवश्यकता होती है। सिस्टम को कई नोड्स में स्केलेबल डेटा संग्रह के लिए डिज़ाइन किया गया है, जो डुप्लिकेट कार्य को रोकने के लिए डिस्ट्रीब्यूटेड टास्क सिंक्रोनाइज़ेशन और साझा कैश का उपयोग करता है। यह यूजर एजेंट और व्यूपोर्ट आयामों को कॉन्फ़िगर करके विशिष्ट क्लाइंट वातावरण का अनुकरण करने की क्षमता के माध्यम से खुद को अलग करता है, जबकि पेज स्क्रीनशॉट जैसे विज़ुअल साक्ष्य कैप्चर करता है।

यह प्रोजेक्ट एक डिस्ट्रीब्यूटेड वेब क्रॉलिंग फ्रेमवर्क है जो स्क्रैपिंग कार्यों की हॉरिजॉन्टल स्केलिंग को सक्षम बनाता है। यह क्रॉल प्रगति और कई सर्वर इंस्टेंस में अनुरोध मेटाडेटा को समन्वयित करने के लिए Redis का उपयोग एक सेंट्रलाइज्ड अनुरोध कतार प्रबंधक और स्टेट स्टोर के रूप में करता है। सिस्टम एक एकल अनुरोध कतार साझा करके क्रॉलिंग वर्कलोड को वितरित करता है और कई श्रमिकों को एक ही पेज पर जाने से रोकने के लिए एक डिस्ट्रीब्यूटेड डुप्लिकेट फिल्टर का उपयोग करता है। यह साझा रिमोट स्टोर के भीतर JSON स्ट्रिंग्स के रूप में जटिल अनुरोध स्थिति और मेटाडेटा को बनाए रखता है।

SparkInternals is a technical reference and architecture guide detailing the internal design and implementation of the Apache Spark distributed computing engine. It serves as a study of big data engine analysis, focusing on how the system manages cluster execution and the interaction between driver nodes, executors, and workers. The project provides a detailed breakdown of how logical plans are converted into physical execution stages. It specifically analyzes the mechanics of data shuffle operations, memory management, and the coordination of distributed job scheduling. The documentation co