How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
Dagster is a data orchestration platform designed to manage the entire lifecycle of data assets through declarative modeling and version-controlled code. It functions as a workflow engine that treats data assets as first-class primitives, allowing teams to define, schedule, and monitor complex pipelines while maintaining clear visibility into lineage, dependencies, and data quality. The platform distinguishes itself by using a code-as-configuration framework that enables standard software engineering practices, such as unit testing and local mocking, to be applied directly to data workflows.
🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines
Unified Interface for Constructing and Managing Workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.
Luigi is a Python framework designed for building and managing complex batch data pipelines. It functions as a workflow orchestration engine that organizes tasks into directed acyclic graphs, ensuring that jobs execute in the correct logical order based on their dependencies. By utilizing a centralized scheduler, the system coordinates task execution across distributed environments, tracks global workflow state, and prevents redundant processing by verifying the existence of output targets before triggering any work. The project distinguishes itself through a robust state-tracking mechanism t
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
The main features of dagworks-inc/hamilton are: Workflow Orchestration, Data Manipulation, Data Pipelines, Data Quality and Observability, Open Source Catalogs, Orchestration Tools, Workflow Frameworks.
Projects with overlapping indexed features include: spotify/luigi — Luigi is a Python framework designed for building and managing complex batch data pipelines. It functions as a… couler-proj/couler — Unified Interface for Constructing and Managing Workflows on different workflow engines, such as Argo Workflows,… dagster-io/dagster — Dagster is a data orchestration platform designed to manage the entire lifecycle of data assets through declarative… dotflow-io/dotflow — 🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines. cocoindex-io/cocoindex — Cocoindex is an incremental data processing engine that builds and maintains live indexes for AI agents, with a core… argoproj/argo-workflows — Argo Workflows is a container-native workflow engine that functions as a Kubernetes custom resource controller. It…