Modern Data Asset Orchestrators

Kedro is a data science pipeline framework and production toolbox designed to build reproducible, modular workflows using software engineering best practices. It functions as a data engineering orchestrator and catalog manager, bridging the gap between interactive analysis and maintainable production pipelines. The framework distinguishes itself by using a data catalog to decouple data access from processing logic and providing tools to transition analysis from interactive notebooks into structured workflows. It includes a workflow visualization tool that generates visual maps of data pipelin

Kedro is a data science pipeline framework and orchestration tool designed to build reproducible and modular data engineering workflows. It functions as an MLOps project template and Python data workflow tool that enforces software engineering best practices to move projects from prototype to production. The system distinguishes itself through a centralized data catalog manager that abstracts data access and versioning across various file formats and cloud storage systems. It further separates processing logic from data access via a lazy-loading data registry and provides a standardized proje

ZenML is an extensible machine learning orchestration framework designed to manage the end-to-end lifecycle of data pipelines and AI agent workflows. It functions as a durable orchestrator that executes machine learning tasks as directed acyclic graphs, ensuring that every step is containerized for consistent performance across local, cloud, and hybrid infrastructure. By decoupling pipeline code from underlying compute and storage backends, the platform allows developers to define infrastructure-agnostic stacks that remain portable across diverse environments. The project distinguishes itself

Dagster is a data orchestration platform designed to manage the entire lifecycle of data assets through declarative modeling and version-controlled code. It functions as a workflow engine that treats data assets as first-class primitives, allowing teams to define, schedule, and monitor complex pipelines while maintaining clear visibility into lineage, dependencies, and data quality. The platform distinguishes itself by using a code-as-configuration framework that enables standard software engineering practices, such as unit testing and local mocking, to be applied directly to data workflows.

Flyte is a Kubernetes-based machine learning orchestrator and containerized pipeline manager designed for coordinating AI workflows and data pipelines. It functions as an engine for defining and executing resilient pipelines, utilizing a data lineage tracker to maintain immutable execution states and ensure reproducible outputs. The platform distinguishes itself by packaging individual tasks into separate containers to ensure dependency isolation and environment consistency. It provides specialized capabilities for machine learning, including the transformation of trained models into scalable

Taipy is a Python data application framework designed for building production-ready web interfaces that integrate AI algorithms and data pipelines. It provides a comprehensive suite for developing AI dashboards and data application interfaces. The framework includes a data pipeline orchestrator for managing complex workflows and dependencies, alongside a job scheduler for executing background tasks on recurring timetables. It also features a scenario analysis tool for performing what-if simulations by comparing different data input versions to evaluate their impact on algorithm outcomes. Add

DVC is a data versioning tool and pipeline orchestrator designed to track large datasets and machine learning models. It functions as a system for managing large data artifacts by storing lightweight metadata in version control while keeping the actual binaries in a separate cache. The project serves as an experiment tracker and remote storage synchronizer, enabling the execution and comparison of machine learning iterations based on hyperparameters and performance metrics. It provides a bridge for pushing and pulling these large data artifacts between local environments and cloud or on-premi

DataHub is a metadata management system and data catalog platform designed to provide a centralized directory for discovering, managing, and documenting datasets across a diverse data stack. It serves as a comprehensive framework for metadata management, incorporating a data governance framework to classify sensitive information and assign ownership for organizational accountability. The platform distinguishes itself through AI-enabled data discovery, which connects large language models to a metadata graph to allow for natural language search and exploration of data assets. It also provides

This is a scheduling library for executing recurring tasks in Go applications using cron-style timing expressions. It provides a programmatic interface to trigger functions at specific intervals based on standard calendar patterns. The system manages periodic job execution by converting timing strings into rules that determine execution times. It uses a job interface to wrap custom logic, allowing for the automation of repetitive background tasks and data synchronization within a Go runtime.

node-schedule is a job scheduler for Node.js that executes arbitrary functions based on specific dates or recurring rules. It functions as a date-based and event-driven scheduling system for JavaScript environments, allowing for the automation of one-off events and periodic tasks. The project supports recurring job scheduling through cron-style strings and recurrence rules, as well as the ability to schedule tasks for precise, single execution dates. It includes timezone-aware date calculations to map recurring rules against global or local time offsets. The system provides a comprehensive j

Amundsen is a data catalog and discovery platform that provides a centralized directory for indexing tables and dashboards. It functions as a metadata management system and search engine, allowing users to locate and understand available data assets across diverse distributed sources. The platform includes capabilities for data lineage tracking to map the origin and movement of datasets between systems. It also serves as a data profiling tool, calculating distribution and quality statistics for individual table columns to provide automated insights into the nature of the data. The system man

APScheduler is a Python task scheduler designed to execute functions at specific times or recurring intervals. It functions as an asynchronous background scheduler and distributed job dispatcher, allowing tasks to run concurrently with application lifecycles and web server request handling. The system distinguishes itself through a persistent job store that saves schedules and task states in external databases, ensuring continuity across process restarts. It separates task scheduling from execution by dispatching jobs to distributed workers in separate processes to prevent execution bottlenec