awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com

Machine Learning Pipeline Orchestrators

Ranking updated Jun 30, 2026

For an orchestrator for ML pipelines, the first results are maiot-io/zenml (ZenML is an ML orchestration framework that defines and executes pipelines as DAGs with containerized steps, infrastructure portability, and built-in monitoring and logging, making it a comprehensive fit for scheduling, managing, and reproducing ML workflows at scale), apache/airflow (Apache Airflow is a mature workflow orchestrator that lets you define ML pipelines as DAGs, schedule them, monitor execution, and scale across distributed workers—exactly matching the need for reproducible and monitored ML pipeline orchestration) and netflix/metaflow (Metaflow is a purpose-built Python framework for defining ML pipelines as DAGs with step-based workflows, automatic checkpointing for reproducibility, and integration with cloud compute—directly matching the need for scheduling, monitoring, and distributed execution). prefecthq/prefect and flyteorg/flyte round out the shortlist. Compare the match explanations and check the project documentation against your requirements.

Open-source workflow engines designed to define, schedule, and monitor complex machine learning data processing pipelines.

Machine Learning Pipeline Orchestrators

Find the best repos with AI.We'll search the best matching repositories with AI.
  • maiot-io/zenmlmaiot-io avatar

    maiot-io/zenml

    5,452View on GitHub↗

    ZenML is an extensible machine learning orchestration framework designed to manage the end-to-end lifecycle of data pipelines and AI agent workflows. It functions as a durable orchestrator that executes machine learning tasks as directed acyclic graphs, ensuring that every step is containerized for consistent performance across local, cloud, and hybrid infrastructure. By decoupling pipeline code from underlying compute and storage backends, the platform allows developers to define infrastructure-agnostic stacks that remain portable across diverse environments. The project distinguishes itself

    ZenML is an ML orchestration framework that defines and executes pipelines as DAGs with containerized steps, infrastructure portability, and built-in monitoring and logging, making it a comprehensive fit for scheduling, managing, and reproducing ML workflows at scale.

    PythonML Workflow OrchestrationsModular Pipeline OrchestratorsExecution Logs
    View on GitHub↗5,452
  • apache/airflowapache avatar

    apache/airflow

    47,414View on GitHub↗

    Airflow is a platform for programmatically authoring, scheduling, and monitoring complex data pipelines. It functions as a workflow automation engine that manages the lifecycle of recurring business processes by executing code-defined task dependencies. By representing workflows as directed acyclic graphs, the system ensures that task execution order and data flow are explicitly defined and reliably maintained across distributed computing environments. The platform distinguishes itself through a highly modular, provider-based architecture that decouples core orchestration logic from external

    Apache Airflow is a mature workflow orchestrator that lets you define ML pipelines as DAGs, schedule them, monitor execution, and scale across distributed workers—exactly matching the need for reproducible and monitored ML pipeline orchestration.

    PythonPipeline Monitoring Dashboards
    View on GitHub↗47,414
  • netflix/metaflowNetflix avatar

    Netflix/metaflow

    10,341View on GitHub↗

    Metaflow is a Python machine learning framework and MLOps workflow orchestrator designed to manage the lifecycle of data pipelines from local prototyping to production. It serves as a distributed compute manager and an experiment tracking system, enabling the creation of reproducible pipelines that transition between development and high-availability production environments. The framework distinguishes itself through an integrated checkpointing system that automatically persists intermediate data artifacts to remote storage, allowing failed runs to be resumed from the last successful step. It

    Metaflow is a purpose-built Python framework for defining ML pipelines as DAGs with step-based workflows, automatic checkpointing for reproducibility, and integration with cloud compute—directly matching the need for scheduling, monitoring, and distributed execution.

    PythonDependency-Isolated
    View on GitHub↗10,341
  • prefecthq/prefectPrefectHQ avatar

    PrefectHQ/prefect

    24,130View on GitHub↗

    Prefect is a workflow orchestration platform designed to define, schedule, and monitor complex data pipelines as Python code. It functions as a container-native engine that wraps individual tasks in isolated environments, ensuring consistent dependencies and resource allocation across diverse infrastructure. By utilizing a state-machine-based orchestration model, the system tracks execution progress through discrete transitions and persistent event logs to maintain reliable and observable task processing. The platform distinguishes itself through a decoupled worker-API architecture, which sep

    Prefect is a workflow orchestration platform that lets you define, schedule, and monitor ML pipelines as DAGs in Python, with built-in observability, task isolation for reproducibility, and a distributed worker architecture that scales across infrastructure.

    PythonExecution Logs
    View on GitHub↗24,130
  • flyteorg/flyteflyteorg avatar

    flyteorg/flyte

    7,095View on GitHub↗

    Flyte is a Kubernetes-based machine learning orchestrator and containerized pipeline manager designed for coordinating AI workflows and data pipelines. It functions as an engine for defining and executing resilient pipelines, utilizing a data lineage tracker to maintain immutable execution states and ensure reproducible outputs. The platform distinguishes itself by packaging individual tasks into separate containers to ensure dependency isolation and environment consistency. It provides specialized capabilities for machine learning, including the transformation of trained models into scalable

    Flyte is a Kubernetes-native ML orchestrator that lets you define DAG pipelines with containerized tasks, provides scheduling, monitoring via a data lineage tracker, and ensures reproducibility and scalable distributed execution, exactly matching your requirements.

    GoData Pipeline AutomationWorkflow OrchestrationAI Workflow Orchestrators
    View on GitHub↗7,095
  • zenml-io/zenmlzenml-io avatar

    zenml-io/zenml

    5,637View on GitHub↗

    ZenML is an orchestration platform designed for building, deploying, and monitoring reproducible machine learning pipelines and agentic workflows. It provides a unified framework that manages the entire lifecycle of machine learning assets, from data processing and model training to the deployment of persistent inference services. By decoupling pipeline logic from underlying compute and storage, the platform enables teams to transition workflows seamlessly from local development environments to production-grade cloud infrastructure. The platform distinguishes itself through a service-oriented

    ZenML is an ML pipeline orchestrator that natively supports DAG-based pipeline definition, scheduling, monitoring, and reproducibility with first-class machine-learning integration, making it a comprehensive fit for this search.

    PythonWorkflow VersioningMachine Learning Workflow LibrariesModular Pipeline Orchestrators
    View on GitHub↗5,637
  • kubeflow/pipelineskubeflow avatar

    kubeflow/pipelines

    4,154View on GitHub↗

    This project is a containerized machine learning workflow engine and orchestrator designed to automate the end-to-end lifecycle of machine learning models on Kubernetes clusters. It functions as an MLOps pipeline compiler that transforms a domain-specific language into structured specifications for portable and scalable deployment. The platform provides a multi-tenant environment with isolated namespaces and identity provider authentication. It distinguishes itself through a combination of container-based task isolation, strongly typed artifact management for data passing, and content-address

    Kubeflow Pipelines is a dedicated ML pipeline orchestration platform that provides DAG-based pipeline definition, scheduling, monitoring, and scalable execution on Kubernetes, aligning perfectly with the need for a tool to define, schedule, and monitor ML workflows.

    PythonML Workflow OrchestrationsPipeline Version Storages
    View on GitHub↗4,154
  • polyaxon/polyaxonpolyaxon avatar

    polyaxon/polyaxon

    3,746View on GitHub↗

    Polyaxon is a Kubernetes-native machine learning orchestration platform and MLOps pipeline orchestrator. It serves as a control plane for managing distributed deep learning workloads, automated machine learning pipelines, and experiment tracking. The platform distinguishes itself through specialized services for distributed training management, including MPI-based coordination for PyTorch and TensorFlow. It provides an automated hyperparameter optimization service utilizing Bayesian, random, and grid search algorithms, alongside managed interactive AI workspaces for launching Jupyter notebook

    Polyaxon is a dedicated ML orchestration platform that lets you define, schedule, and monitor DAG-based pipelines, with built-in support for reproducibility, logging, and distributed execution—matching every core feature of your search.

    MDXML Workflow Orchestrations
    View on GitHub↗3,746
  • clearml/clearmlclearml avatar

    clearml/clearml

    6,934View on GitHub↗

    ClearML is a comprehensive MLOps platform designed to manage the end-to-end machine learning lifecycle, from initial experimentation to production deployment. It provides a suite of integrated tools including a pipeline orchestrator for automating workflows, an experiment tracking tool for logging hyperparameters and metrics, and a metadata-driven data versioning system for managing large-scale datasets and model artifacts. The platform is distinguished by its advanced compute management and serving capabilities. It features a GPU compute manager that supports fractional resource slicing and

    ClearML is an MLOps platform that includes a pipeline orchestrator for automating ML workflows, which fits DAG-based pipeline definition, scheduling, monitoring, and distributed execution, though its primary identity is a broader lifecycle platform rather than a dedicated pipeline engine.

    PythonML Workflow OrchestrationsPipeline Monitoring Dashboards
    View on GitHub↗6,934
  • temporalio/temporaltemporalio avatar

    temporalio/temporal

    24,252View on GitHub↗

    Temporal is a distributed workflow orchestration engine designed to manage fault-tolerant, stateful, and long-running background processes. It functions as a platform for coordinating complex cross-service operations, ensuring consistency and reliability in distributed environments by decoupling workflow orchestration from task execution. The platform distinguishes itself through a deterministic, event-sourced execution model that reconstructs workflow state by re-executing code from an immutable event log. This approach isolates non-deterministic side effects into managed activities, allowin

    Temporal is a general-purpose distributed workflow orchestration engine that can model DAG-like pipelines, schedule tasks, and provide monitoring and logging, but it lacks native integration with machine learning frameworks, so it fits the workflow-management aspect of your search without the ML-specific features you likely need.

    GoWorkflow VersioningExecution Logs
    View on GitHub↗24,252
  • comfy-org/comfyuiComfy-Org avatar

    Comfy-Org/ComfyUI

    145,590View on GitHub↗

    ComfyUI is a node-based generative AI orchestration engine designed for constructing, testing, and executing complex image and video synthesis pipelines. By utilizing a directed acyclic graph execution model, the platform allows users to build reproducible workflows through modular, interconnected processing blocks without requiring manual code implementation. It serves as both a local environment for high-performance model inference and a production-ready server for deploying generative capabilities. The platform distinguishes itself through its focus on workflow portability and extensibilit

    ComfyUI is a node-based generative AI orchestration engine that uses a DAG execution model for building reproducible workflows, fitting the core DAG-based pipeline definition, but it is specialized for generative AI (image/video synthesis) rather than covering general ML pipeline orchestration, scheduling, and monitoring.

    PythonNode-Based Generative PipelinesDirected Acyclic Graph Execution EnginesGenerative AI Orchestration Engines
    View on GitHub↗145,590
  • huggingface/ml-internhuggingface avatar

    huggingface/ml-intern

    10,521View on GitHub↗

    This project is an autonomous AI agent framework and workflow orchestrator designed to automate machine learning engineering. It functions as a reasoning engine that reads research papers and writes code to train and deploy machine learning models through iterative reasoning loops and tool execution. The system distinguishes itself by integrating a GPU-accelerated sandboxed execution environment, allowing it to run and verify machine learning scripts in isolated remote containers. It utilizes a model provider integration gateway to route inference requests across various hosted or local endpo

    This project is an autonomous AI agent framework for automating ML engineering through iterative reasoning loops, not a DAG-based pipeline orchestrator for defining, scheduling, and monitoring task graphs — it lacks the core pipeline scheduling and monitoring features you are looking for.

    PythonMachine Learning Workflow LibrariesML Workflow Orchestrations
    View on GitHub↗10,521
Compare the top 10 at a glance
RepositoryStarsLanguageLicenseLast push
maiot-io/zenml5.5KPythonApache-2.0Jun 17, 2026
apache/airflow47.4KPythonApache-2.0Nov 7, 2026
netflix/metaflow10.3KPythonApache-2.0Nov 6, 2026
prefecthq/prefect24.1KPythonApache-2.0Nov 7, 2026
flyteorg/flyte7.1KGoApache-2.0Jun 17, 2026
zenml-io/zenml5.6KPythonApache-2.0Nov 7, 2026
kubeflow/pipelines4.2KPythonApache-2.0Jun 17, 2026
polyaxon/polyaxon3.7KMDXApache-2.0Nov 7, 2026
clearml/clearml6.9KPythonApache-2.0Nov 3, 2026
temporalio/temporal24.3KGoMITNov 7, 2026

Related searches

  • a workflow orchestration tool for data pipelines
  • an embeddable workflow engine for orchestrating pipelines
  • a self-hosted Airflow alternative
  • a Python framework for data pipelines
  • Pipelines, ETL/ELT and orchestration
  • a platform for managing the ML lifecycle
  • a framework for building scalable data pipelines
  • a visual builder for AI agent pipelines