Awesome GitHub RepositoriesBatch Processing Schedulers

Airflow is a platform for programmatically authoring, scheduling, and monitoring complex data pipelines. It functions as a workflow automation engine that manages the lifecycle of recurring business processes by executing code-defined task dependencies. By representing workflows as directed acyclic graphs, the system ensures that task execution order and data flow are explicitly defined and reliably maintained across distributed computing environments. The platform distinguishes itself through a highly modular, provider-based architecture that decouples core orchestration logic from external

Luigi is a Python framework designed for building and managing complex batch data pipelines. It functions as a workflow orchestration engine that organizes tasks into directed acyclic graphs, ensuring that jobs execute in the correct logical order based on their dependencies. By utilizing a centralized scheduler, the system coordinates task execution across distributed environments, tracks global workflow state, and prevents redundant processing by verifying the existence of output targets before triggering any work. The project distinguishes itself through a robust state-tracking mechanism t

Argo is a cloud native CI/CD platform and Kubernetes workflow engine. It functions as a container pipeline orchestrator and job scheduler, managing multi-step sequences of containers as jobs using directed acyclic graphs within a cluster. The system acts as a progressive delivery controller, reducing release risk through automated Canary and Blue-Green deployment strategies. It provides declarative GitOps synchronization to mirror the state of a git repository directly into the cluster environment for continuous delivery automation. The platform covers a broad range of capabilities including

Argo Workflows is a container-native workflow engine that functions as a Kubernetes custom resource controller. It orchestrates complex sequences of containerized tasks by executing them as directed acyclic graphs, allowing for dependency management and parallel processing within a cluster. The system extends the native Kubernetes control plane to manage the full lifecycle of automated processes, from initial triggering to final resource cleanup. The platform distinguishes itself through its controller-pattern reconciliation, which continuously monitors workflow states to align them with desi

Nomad is a distributed workload orchestrator and infrastructure automation platform designed to manage the lifecycle of applications across large-scale, heterogeneous environments. It functions as a multi-cloud orchestration engine, providing a unified control plane to deploy, scale, and govern containers, virtual machines, and legacy applications. By utilizing declarative job specifications, the system ensures infrastructure convergence and maintains the desired state across distributed data centers and geographic regions. The platform distinguishes itself through a flexible, plugin-based ar

Unstructured is an enterprise-grade data orchestration engine designed to transform raw, unstructured files into structured, machine-readable formats. It functions as a comprehensive platform for document ingestion, partitioning, and enrichment, specifically engineered to prepare complex data for retrieval-augmented generation and agentic AI workflows. The platform distinguishes itself through its sophisticated document processing strategies, which combine rule-based extraction with vision-language models to handle diverse file layouts, tables, and images. It provides a modular architecture t

Dask 是一个并行计算框架和分布式任务调度器，旨在将 Python 数据科学工作流从单机扩展到大型集群。它作为一个集群资源管理器，通过将任务及其依赖项表示为有向无环图来编排计算逻辑。这种架构允许系统在管理复杂执行要求的同时，自动将工作负载分配到可用硬件上。 该项目通过一个延迟评估引擎脱颖而出，该引擎将数据操作推迟到明确请求时才执行，从而实现全局图优化和高效的资源分配。它结合了内存感知数据溢出功能，以防止在处理超过可用内存的数据集时系统崩溃，并利用任务图融合将操作序列组合成单个执行步骤，从而最大限度地减少调度开销和节点间通信。 该平台为大规模数据分析提供了全面的功能面，包括对分布式机器学习、高性能计算集成和并行数据处理的支持。它提供了用于集群生命周期管理、性能分析和任务执行实时监控的广泛工具。用户可以在各种基础设施上部署这些环境，包括本地硬件、云提供商、容器化系统和高性能计算集群。

DataLoader is a utility that collects individual data loads into a single batch and caches results to minimize redundant backend requests. It operates on a batch-and-cache architecture, where multiple data lookups within a single execution frame are grouped together and dispatched as one request, with the results stored in memory for instant retrieval on subsequent calls. The utility distinguishes itself through several key capabilities. It supports per-key error handling, allowing partial failures within a batch without rejecting the entire operation. A cache priming mechanism lets developer

Banana-slides is a generative AI workflow engine designed to automate the creation and refinement of professional slide decks. By leveraging large language models, the platform transforms raw text, structured outlines, and existing documents into visual presentations. It functions as an automated tool that orchestrates the entire lifecycle of a presentation, from initial content generation and layout design to final export. The system distinguishes itself through a modular provider abstraction that allows users to integrate various artificial intelligence services for content and image synthe

This tool is a command-line utility designed to synchronize and archive media from cloud storage to local directories. It functions as an automated backup service that maintains a local mirror of remote photo libraries, ensuring that local storage remains current with remote changes through periodic monitoring and incremental updates. The project distinguishes itself through its support for persistent, containerized background execution, which allows for continuous, automated management of media collections. It provides robust multi-account isolation, enabling users to manage multiple indepen

Feast is an open-source feature store for machine learning that provides a central platform for defining, storing, and serving features across both training and inference workflows. It operates as a declarative system where feature definitions are written as code in Python files, synchronized to a central registry, and made available for low-latency online retrieval or point-in-time correct historical joins for training datasets. The project abstracts storage behind a pluggable architecture, allowing offline and online backends to be swapped without changing retrieval logic, and coordinates ma

Qor 是一个 Go 管理框架和后端工具包，用于构建管理界面、无头内容管理系统和 REST API 生成器。它提供了一个用于实现业务应用程序后端的结构化环境，专门从事结构化内容和媒体资产的管理。 该项目通过全面的多语言内容管理脱颖而出，具有基于区域设置的数据版本控制以及用于国际化和翻译管理的专用系统。它进一步通过内置的用于业务流程自动化的状态机实现和用于在发布前审查更改的内容暂存工作流来区分其产品。 该框架涵盖了广泛的功能，包括基于角色的访问控制、会话管理和后台作业调度。其数据管理面包括 CRUD 处理程序覆盖、关系管理以及基于后端资源定义生成仪表板和表单输入的元数据驱动 UI。此外，它还提供用于 RESTful API 生成的工具，支持内容协商和嵌套端点。 该系统允许通过将 HTML 模板直接编译到 Go 应用程序二进制文件中来优化部署，从而消除对文件系统的依赖。

这是一个自动化数字内容领取和游戏商店自动化机器人。它作为一个无头客户端，处理账户认证和请求序列，按计划收集免费数字游戏和可下载内容。 该工具为 Epic Games Store、GOG 和 Amazon Prime Gaming 提供特定的自动化功能。它使用商店特定的适配器逻辑来锁定限时优惠，并在无需手动干预浏览器的情况下构建数字游戏库。 系统集成了基于 cron 的任务调度（用于每日检查）、使用存储凭据的自动登录流程以及无头浏览器自动化。它还包含一个通知系统，通过外部 Webhook 发送领取状态提醒。

Orchest 是一个数据流水线编排器和容器化工作流管理器。它提供了一个平台，通过图形界面和脚本的结合来设计、调度和执行复杂的数据处理序列。 该平台通过使用容器管理软件依赖项而脱颖而出，确保了跨不同环境的一致执行。它具有一个多语言任务调度器，能够触发用多种编程语言编写的作业，并包含一个跟踪项目配置和代码历史快照的版本控制系统。 系统涵盖了可视化工作流设计和基于图的依赖映射，以及用于循环或即时执行的时间触发任务调度。它还支持部署在流水线运行期间保持活跃的持久化后台服务。

Panda Factor is a quantitative trading infrastructure and alpha factor framework. It serves as a backend system for building, calculating, and managing mathematical signals designed to predict the price movements of financial assets. The project functions as a technical indicator engine that generates quantitative metrics from price and volume data. It utilizes a financial data pipeline to automate the synchronization of market data from multiple providers on a nightly schedule. The system provides capabilities for quantitative alpha generation and the construction of financial indicators us