How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
Dagster is a data orchestration platform designed to manage the entire lifecycle of data assets through declarative modeling and version-controlled code. It functions as a workflow engine that treats data assets as first-class primitives, allowing teams to define, schedule, and monitor complex pipelines while maintaining clear visibility into lineage, dependencies, and data quality. The platform distinguishes itself by using a code-as-configuration framework that enables standard software engineering practices, such as unit testing and local mocking, to be applied directly to data workflows.
Unified Interface for Constructing and Managing Workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.
Kestra is a declarative workflow orchestrator designed to manage complex task dependencies and automated processes through versioned configuration files. It functions as a distributed platform that decouples task scheduling from execution by offloading computational workloads to a fleet of worker nodes. The system uses a reactive, event-driven engine to initiate workflows automatically in response to external signals, webhooks, schedules, or file system changes. The platform distinguishes itself through a modular plugin architecture that allows for the integration of custom tasks and external
🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines
The main features of dotflow-io/dotflow are: Workflow Orchestration, Workflow Frameworks.
Projects with overlapping indexed features include: dagworks-inc/hamilton — Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode… pinterest/pinball — Pinball is a scalable workflow manager. couler-proj/couler — Unified Interface for Constructing and Managing Workflows on different workflow engines, such as Argo Workflows,… dagster-io/dagster — Dagster is a data orchestration platform designed to manage the entire lifecycle of data assets through declarative… kestra-io/kestra — Kestra is a declarative workflow orchestrator designed to manage complex task dependencies and automated processes… quantumblacklabs/kedro — Kedro is a data science pipeline framework and production toolbox designed to build reproducible, modular workflows…