Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
Dagster is a data orchestration platform designed to manage the entire lifecycle of data assets through declarative modeling and version-controlled code. It functions as a workflow engine that treats data assets as first-class primitives, allowing teams to define, schedule, and monitor complex pipelines while maintaining clear visibility into lineage, dependencies, and data quality. The platform distinguishes itself by using a code-as-configuration framework that enables standard software engineering practices, such as unit testing and local mocking, to be applied directly to data workflows.
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
Airflow is a platform for programmatically authoring, scheduling, and monitoring complex data pipelines. It functions as a workflow automation engine that manages the lifecycle of recurring business processes by executing code-defined task dependencies. By representing workflows as directed acyclic graphs, the system ensures that task execution order and data flow are explicitly defined and reliably maintained across distributed computing environments. The platform distinguishes itself through a highly modular, provider-based architecture that decouples core orchestration logic from external
Data policy IN, dynamic view OUT: PACE is the Policy As Code Engine. It helps you to programatically create and apply a data policy to a processing platform like Databricks, Snowflake or BigQuery (or plain 'ol Postgres, even!) with definitions imported from Collibra, Datahub, ODD and the like.
The main features of getstrm/pace are: Workflow Orchestration.
Open-source alternatives to getstrm/pace include: apache/airflow — Airflow is a platform for programmatically authoring, scheduling, and monitoring complex data pipelines. It functions… bruin-data/bruin — Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end… dagster-io/dagster — Dagster is a data orchestration platform designed to manage the entire lifecycle of data assets through declarative… dagworks-inc/hamilton — Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode… dotflow-io/dotflow — 🎲 Dotflow turns an idea into flow! — Lightweight Python library for execution pipelines. kestra-io/kestra — Kestra is a declarative workflow orchestrator designed to manage complex task dependencies and automated processes…