Awesome GitHub RepositoriesDAG-Based Dependency Resolution

Airflow is a workflow orchestration platform for authoring, scheduling, and monitoring complex data pipelines as code using Python. It employs a DAG-based task scheduler to manage execution timing and dependencies via directed acyclic graphs, utilizing a distributed task execution engine to run workloads across a cluster of worker nodes. The platform provides a data pipeline monitor for tracking the health and execution history of programmatic workflows. This includes a web interface for workflow progress visualization and health monitoring to identify and troubleshoot pipeline failures. The

This project is a Python workflow orchestration platform and programmatic data pipeline engine used to author, schedule, and monitor complex data pipelines. It functions as a directed acyclic graph manager and scheduler, allowing users to define data movement and transformation tasks as code to ensure precise execution order and maintainability. The platform distinguishes itself by treating workflows as code, enabling pipelines to be versioned and tested through a standard programming language. It utilizes a system of extensible operators to encapsulate integration logic and employs a templat

DVC is a data versioning tool and pipeline orchestrator designed to track large datasets and machine learning models. It functions as a system for managing large data artifacts by storing lightweight metadata in version control while keeping the actual binaries in a separate cache. The project serves as an experiment tracker and remote storage synchronizer, enabling the execution and comparison of machine learning iterations based on hyperparameters and performance metrics. It provides a bridge for pushing and pulling these large data artifacts between local environments and cloud or on-premi

Kedro is a data science pipeline framework and orchestration tool designed to build reproducible and modular data engineering workflows. It functions as an MLOps project template and Python data workflow tool that enforces software engineering best practices to move projects from prototype to production. The system distinguishes itself through a centralized data catalog manager that abstracts data access and versioning across various file formats and cloud storage systems. It further separates processing logic from data access via a lazy-loading data registry and provides a standardized proje

Metaflow is a Python machine learning framework and MLOps workflow orchestrator designed to manage the lifecycle of data pipelines from local prototyping to production. It serves as a distributed compute manager and an experiment tracking system, enabling the creation of reproducible pipelines that transition between development and high-availability production environments. The framework distinguishes itself through an integrated checkpointing system that automatically persists intermediate data artifacts to remote storage, allowing failed runs to be resumed from the last successful step. It

Enterprise job scheduling middleware with distributed computing ability.

Hatchet is an open-source durable workflow engine and task orchestration platform. It provides a framework for building and executing fault-tolerant, multi-step pipelines as directed acyclic graphs (DAGs), with automatic retries, scheduling, and real-time observability. The system is built around durable task checkpointing, which persists execution state after each step so work can resume from the last checkpoint after a worker crash or restart, and it supports event-driven task resumption that pauses a task until a matching external event arrives. The platform distinguishes itself through it

Open Multi-Agent is a TypeScript framework for multi-agent orchestration that decomposes natural language goals into a runtime-generated directed acyclic graph of tasks. It functions as a task orchestrator and workflow state manager, coordinating multiple AI models to execute parallel and sequential operations. The framework is distinguished by a proposer-judge consensus protocol used to validate agent outputs through a quorum of agreement. It employs provider-agnostic model routing to assign specific models to tasks based on roles or execution phases and utilizes state-based workflow checkpo

Osmedeus is a security workflow orchestration engine that coordinates AI agents, shell commands, and scanning tools through declarative YAML pipelines. It functions as a distributed security scanner, a declarative workflow automator, and an AI agent framework for security, enabling automated multi-step security analysis with conditional branching, parallel execution, and distributed workers. The engine distinguishes itself through a hybrid runner model that executes workflow steps on the local host, inside Docker containers, or over SSH to remote machines, selected per step or module. It supp

Volcano is a Kubernetes-native batch scheduler specialized for AI, machine learning, and high-performance computing workloads. It provides gang scheduling to atomically allocate resources for all tasks of a distributed job, preventing deadlocks from partial allocation, and supports hierarchical queue management for multi-tenant resource isolation with configurable quotas, borrowing, and preemption. Topology-aware placement optimizes communication-intensive workloads by modeling network hierarchy to minimize cross-switch latency. Volcano differentiates itself with automated orchestration of di

Azkaban 是一个分布式工作流管理器和基于 DAG 的作业编排器，设计为企业级批处理程序。它作为一个基于 Java 的工作流引擎，跨执行器服务器集群调度和执行复杂的作业序列，并具有管理 Hadoop 集群上大数据工作负载的特定功能。 该系统通过分布式执行器模型脱颖而出，该模型通过共享数据库协调状态以确保高可用性。它采用基于插件的架构，允许自定义作业类型和系统功能扩展，包括在不重启执行器服务器的情况下热重载插件的能力。 该平台涵盖了广泛的功能，包括具有条件逻辑的数据管道编排、定期和事件驱动的调度，以及具有 SLA 跟踪的企业监控。它提供细粒度的访问控制和用户模拟以实现安全执行，以及用于执行器负载均衡和资源配额的流量管理工具。 用户可以通过基于 Web 的界面或以编程方式通过工作流执行 API 管理工作流。

ms-agent is an LLM agent framework and multi-agent orchestration system designed to build autonomous entities that combine large language models with tool calling and structured workflows. It serves as a tool integration platform and workflow engine for executing complex tasks through the coordination of specialized agents. The project distinguishes itself through a multimodal agent workflow engine capable of automating the production of text, images, and video. It features a sandboxed code execution environment for running generated code and quantitative data analysis in isolated containers,

OpenSquilla 是一个 LLM 智能体编排框架，旨在利用有向无环图协调多步 AI 工作流和工具执行。它作为一个集中式系统，用于管理专门的技能包并执行复杂的推理序列。 该项目通过一个路由网关脱颖而出，该网关根据复杂性、成本和性能将任务定向到不同的 AI 提供商。它利用多层 AI 记忆系统，通过本地嵌入和 SQLite 组织工作、情景和语义知识，并配有一个安全执行沙盒，通过基于风险的权限配置文件隔离智能体生成的代码。 该平台涵盖了广泛的功能，包括多渠道部署到 Web 和消息平台、通过 cron 进行自动任务调度，以及用于连接外部工具的 Model Context Protocol 网桥。它还提供全面的监控和可观测性工具，用于跟踪 Token 成本、审计运行时决策以及管理可重用技能目录。 该系统包括用于工作区初始化和技能生命周期管理的命令行工具。

该项目是一个 LLM 浏览器自动化框架和 AI 代理浏览器界面。它作为一个控制层，使用大型语言模型将自然语言指令转换为浏览器交互，使 AI 代理能够通过标准化的浏览器控制功能导航并与网页交互。 该系统作为一个 RPA 工作流编排器和无头浏览器管理工具，能够记录和重放确定性浏览器序列以自动化重复任务。它通过隐身配置（包括住宅代理和修改后的浏览器引擎）来绕过机器人检测并解决验证码，从而脱颖而出。 该平台涵盖了广泛的能力，包括结构化 Web 数据提取、用于维护认证的持久会话管理，以及针对多因素认证等复杂步骤的人机协同干预。它支持本地连接和托管云沙盒部署，通过交互式图表提供视觉工作流管理和实时活动监控。 集成通过命令行界面和用于外部 LLM 提供商及第三方编排平台的 API 连接提供。

EFCore.BulkExtensions 是一个用于在 Entity Framework Core 生态系统中执行高性能批量插入、更新和删除操作的库。它充当数据库批量处理工具包，并作为原生 SQL Bulk Copy 的包装器，以实现跨多个数据库提供商的更快数据摄取和同步。 该库为关系数据同步提供了专门的功能，允许用户通过批量更新插入 (upsert) 和条件同步将数据库表与本地实体列表对齐。它还支持关系数据图插入，这使得在维护外键关系的同时，能够跨多个表添加大量父子记录。 该工具涵盖了广泛的批量数据操作，包括常规 CRUD 处理、表截断和直接表对表数据复制。它还包括用于服务器端批量更新和高性能数据库迁移的实用程序，无需将实体加载到应用程序内存中即可移动大量数据。

Polyaxon is a Kubernetes-native machine learning orchestration platform and MLOps pipeline orchestrator. It serves as a control plane for managing distributed deep learning workloads, automated machine learning pipelines, and experiment tracking. The platform distinguishes itself through specialized services for distributed training management, including MPI-based coordination for PyTorch and TensorFlow. It provides an automated hyperparameter optimization service utilizing Bayesian, random, and grid search algorithms, alongside managed interactive AI workspaces for launching Jupyter notebook

Nuke is a build automation system for defining software compilation and deployment pipelines using a strongly typed C# console application. It functions as a cross-platform build engine and pipeline orchestrator that treats build configurations as standard executable programs rather than static files. By leveraging a compiled language, the system provides type safety and IDE support for build script logic. This approach allows for the definition of automation and CI/CD pipelines using a professional programming language instead of YAML or shell scripts. The engine manages .NET project orches

Dag-factory is a framework for constructing and managing Apache Airflow data pipelines through declarative configuration files. By replacing manual procedural code with structured YAML definitions, it enables the programmatic generation of complex workflow structures, task dependencies, and execution schedules. The project distinguishes itself by mapping configuration keys directly to Python class constructors and operators, allowing for the dynamic instantiation of objects and custom logic. It supports hierarchical configuration inheritance to standardize settings across environments and pro